Performance Benchmark Fundamentals
The goal of performance measurement is to understand the performance capabilities and limitations of a system. Every system has limitations, and benchmarks characterize the system in a way that you can understand these limitations.
Benchmarks can be complicated if the system capabilities and limitations vary depending on the demands placed on the system. They also vary based on the resources that are available to the system, for example, CPU, memory, network bandwidth, and disk bandwidth. The set of benchmark measurements must be carefully designed so that the impact of these factors can be understood clearly .
In the above example, the X axis characterizes the input rate, and the Y axis represents the output rate. The system is exposed to load at a controlled rate and is in a steady-state for some period of time. After an initial stabilization period, both the input and output rates are measured, providing one data point for the performance curve. This example assumes that all other factors are being held constant.
The shape of the performance curve tells us a lot about the system under test. If each input results in a single output, then over the normal operating range, the output rate will exactly match the input rate and within statistical bounds. If each input results in more than one output, or if you are measuring data rates instead of input and output rates, there may be a scale factor relating the two values. However, over the normal operating range there should be a linear relationship between the two rates – that is the curve is a straight line.
The input rate that marks the end of this linear region marks the operating capacity of the system. This may mark the true limits of the system design, or it may indicate that some type of resource limit has been hit. This could be the result of the available physical memory or the bandwidth available on the NIC card or the available CPU cycles getting exhausted. It is important to determine the nature of the limit, as this may indicate ways to alter the environment and increase capacity either by tuning the system or adding resources.
Beyond the operating capacity, further increase in the input rate exceeds the capacity of the system to perform work. Once this occurs, increasing the input rate will no longer produce the same level of increase in the output. The inputs are increasing faster than the system is producing output. If the input rate continues to increase it will reach a point where the output rate begins to decline. The system is taking resources away from completing work and applying them to accepting the inputs.
Operating under a load is inherently unstable. Inputs arrive faster than the work is getting completed, and this results in inputs piling up. This buffer for memory, disk, and messaging system is finite in capacity. At full capacity the system will fail. Thus systems can, at best, operate in the overload mode for short periods of time.