A State-Size Inclusive Approach to Optimizing Stream Processing Applications

Omoregbee, P; Forshaw, M; Thomas, N

doi:10.1007/978-3-031-43185-2_22

A State-Size Inclusive Approach to Optimizing Stream Processing Applications

Lookup NU author(s): Dr Paul Omoregbee, Dr Matthew Forshaw ORCiD, Dr Nigel Thomas

Downloads

Accepted version [.pdf]

Licence

This is the authors' accepted manuscript of a conference proceedings (inc. abstract) that has been published in its final definitive form by Springer , 2023.

For re-use rights please refer to the publisher's terms and conditions.

Abstract

In stream processing applications, accurately measuring a system’s processing capacity is critical for ensuring optimal performance and meeting Service Level Objectives (SLOs). Traditionally, operator throughput has been used as a proxy for the application’s state size, but this approach can be misleading when dealing with window-based applications. In this paper, we explore the impact of window selectivity on the performance of streaming applications, demonstrating how a growing application state can artificially decrease the operators’ throughput, resulting in false positives that could trigger premature scaling-down decisions. To address this problem, we conduct empirical evaluations to assess the relationship between operators’ throughput and state size, showcasing the state size pattern typically does not correspond to the operator’s processing rate in window-based applications. Our findings highlight the importance of considering the state size of the application in performance monitoring and decision-making, particularly in the context of window-based applications.