Measuring (nc)WMS performance

---

1. Introduction
2. Tools
3. Setup
4. Running the test
5. Analysing the results
   1. Open the results in Excel
   2. Overview of the statistics
   3. Analysing latency
   4. Analysing responses-per-second
6. Further monitoring
7. Generating more detailed statistics

---

Introduction

It is of course highly desirable that your Web Map Service should be as responsive as possible. However, changes to data formats and to the WMS code can affect performance. The first step to diagnosing performance issues is to measure them, but measuring the real-world performance of a WMS (or any service) reliably is difficult, but this guide presents one possible method. This guide was written with  ncWMS in mind, but will work for any WMS (WMSs such as ncWMS that run on the Java Virtual Machine will benefit from extra monitoring tooling).

This guide presents one method for monitoring performance from the point of view of a web client, i.e. it treats the WMS as a black box and does not consider the internal workings of the WMS at all. We only consider the GetMap operation: the performance of other operations is of lesser concern in general (GetMap is by far the most frequently-called operation in normal use).

When reporting performance issues to the ncWMS mailing list, it would be highly desirable if you could have run a similar test (if at all possible) and quote your results.

Please contact Jon Blower at j.d.blower [at] reading.ac.uk with any comments, suggestions or queries about this page, or post to the ncWMS mailing list.

Tools

You will need:

• A working (nc)WMS server (naturally)
•  Apache JMeter, a GUI-based tool for creating and running test scripts (can also run scripts in batch mode)
• This XSL file, which transforms the results of JMeter into a format that Excel understands

Setup

1. Start your WMS server and verify that it is working correctly (e.g. for ncWMS, load the Godiva2 client and make sure you can see data). For ncWMS, you will probably wish to switch off the server-side cache so that all images are being generated afresh (unless it's the cache that you want to test of course).
2. (Optionally, start jconsole and point it to your application server. You may wish to skip this if you are concerned that jconsole may be affecting your results.)
3. Start Apache JMeter.
4. Load this script as a starting-point. This generates in random order a number of GetMap requests and fires them off to the server in a number of client threads. Expand the tree on the left-hand side of the JMeter GUI if it is collapsed.
5. Make sure the script is pointing to your server by setting the host name in the property entry that currently reads "localhost:8084".
6. Make sure the path to your WMS is correct (e.g. ncWMS/wms)
7. Adjust the LAYERS parameter in the GetMap configuration
8. You may also wish to edit the image format and the tile size, although the defaults of 256x256 PNG files are sensible.
9. In the "Simple Data Writer", adjust the filename to give the fully-qualified path of the desired results file.
10. You can also adjust the load the client places on the server. In "Thread Group", you can adjust the maximum number of threads, which simulates a number of simultaneous users. You can also adjust the ramp-up period (linear ramp-up from 1 thread/user to the maximum) and the number of loops and set the length of the test either as a number of loops or as a timed test.

Running the test

Select Run->Start (or Ctrl-R) in Apache JMeter. You can wait for completion (if you've set a finite time for the test) or press Ctrl-. to stop the test.

If you select the "Graph Results" or "View Results Tree" viewer in the left-hand tree you can see the results as they come in. Look out for errors - if the test is going very fast then you probably have a configuration problem.

Analysing the results

Open the results in Excel

JMeter produces results in an XML format (you specified the file path and name above). To analyse the results in Excel (or similar), you can transform the results to an HTML table using this stylesheet. Download or copy the stylesheet to the same location as your results file. Open the results file in a simple text editor and insert the following line at the top of the file as shown here:

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet href="logStylesheet.xsl" type="text/xsl"?>  <!-- INSERT THIS LINE -->
<testResults version="1.2">

You can now open the results file in Excel (e.g. in Windows Explorer, right-click the file and select "Open with Excel"). You will be asked to select an XSL transformation. Select the "logStylesheet.xml" and press OK. Say Yes to the next pop-up. (This has been tested only in Excel 2007.) You should see a table of results.

Overview of the statistics

The raw results give, for each request, the time of the start of the request, the loading time (i.e. the time the server was processing the request) and the latency (the time between the request and the server's first response). The download time (load time - latency) is also estimated (this is calculated within the style sheet). The results also show the number of active threads (i.e. "users") and the size of the resulting image.

Note that although the images are always the same size (256x256, unless you've adjusted the test script) the size of the image in bytes may vary considerably depending on the image content, due to compression. This could be controlled for by using an uncompressed image format (currently not supported in ncWMS).

Analysing latency

One useful statistic is the variation of latency with the number of active threads. Latency should increase with the number of simultaneous users - you can easily create a plot in Excel to investigate this.

Analysing responses-per-second

The raw results do not show the number of responses that the server has provided per unit time. One way to elicit this is to create a sliding window over the "start time" column. If the "Start time" column is B, create a new column C. In cell C6 enter the formula =10/(B12-B2), which calculates the average rate of response for those ten responses. Copy this formula down the column and you should have an evolution of the number of responses-per-second over the test.

See this Excel file for an example set of results.

Further monitoring

You can use the  jconsole application to monitor the CPU/memory usage of the server. You could use VirtualVM to produce very detailed instrumented information about the objects that are in use on the server, although this degrades performance severely.

Generating more detailed statistics

This is a "black-box" test and so does not consider the internals of the WMS service. If you have control over the code of the WMS, you could add information to the HTTP response headers from the server that contain information about what has happened internally. These HTTP response headers are recorded by JMeter and can be transformed into information that ends up in the Excel spreadsheet. This project took this approach to discover whether the WMS in question was using caching, and various other useful things. Such debug information could be built into ncWMS if this is useful, although we would need to explore the security implications of doing this.