Injecting realistic burstiness to a traditional client-server benchmark

The design of autonomic systems often relies on representative benchmarks for evaluating system performance and scalability. Despite the fact that experimental observations have established that burstiness is a common workload characteristic that has deleterious effects on user-perceived performance, existing client-server benchmarks do not provide mechanisms for injecting burstiness into the workload. In this paper, we introduce a new methodology for generating workloads that emulate the temporal surge phenomenon in a controllable way, thus provide a mechanism that enables testing and evaluation of client-server system performance under reproducible bursty workloads. This new methodology allows to inject different amounts of burstiness into the arrival stream using the index of dispersion, a single parameter that is as simple to use as a turnable knob.

We exemplify the effectiveness of this new methodology by introducing a new module into the TPC-W, a benchmark that is routinely used for capacity planning of e-commerce systems. This new module injects burstiness into the arrival process of clients in a controllable manner, and hence, enables understanding system performance degradation due to burstiness. Detailed experimentation on a real system shows that this benchmark modification can stress the system under different degrees of burstiness, making a strong case for the usefulness of this modification for capacity planning of autonomic systems.