iQCAR: Inter-Query Contention Analyzer

There are many approaches in use today to either prevent or minimize the impact of inter-query interactions on a shared cluster. Preventive measures often provide query execution isolation at the resource allocation level to guarantee a predictable query performance. Despite these measures, performance issues due to concurrent executions of mixed workloads are a common problem in large scale data processing systems. As a result, answering questions like who is causing my query to slowdown is important to diagnose resource conflicts in a multi-tenant environment for accurate blame attribution. However, accurate analysis of resource contention is challenging owing to a complex cause-effect relationship between resource utilization and runtime of concurrent queries (see Figure 1). For example, when some tasks get delayed because of a high demand for a particular resource (e.g. if they are blocked on CPU), they hold on to other resources (e.g. memory) as well, thus causing contention for other concurrently running queries on the held resources. Based on our user-study experience, this process is non-trivial and tedious, and involves hours of manually debugging through a cycle of query interactions.