Toggle Main Menu Toggle Search

Open Access padlockePrints

On Improving Streaming System Autoscaler Behaviour using Windowing and Weighting Methods

Lookup NU author(s): Dr Stuart Jamieson, Dr Matthew ForshawORCiD

Downloads


Licence

This work is licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0).


Abstract

Distributed stream processing systems experience highly variable workloads. This presents a challenge when provisioning compute to meet the needs of these workloads. Rightsizing systems for peak demand leads to often-unacceptable financial cost, motivating the need for adaptive approaches to meet the needs of changing workloads. The choice of parallelism of workload operators are commonly governed by autoscalers, but their behaviour is often case specific and highly sensitive to the choice of tunable parameters and thresholds. This presents a challenge to practitioners wishing to understand the performance implications of their decisions.We systematically explore the impact of parameter tuning for a state-of-the-art autoscaler; identifying impacts in terms of SASO properties as well as behavioural phenomena such as extreme parallelism shifts and robustness. Autoscalers commonly make decisions on instantaneous system performance, without incorporating historical information. This seeks to mitigate challenges of being overly influenced by historical values, to be able to respond in response to the evolving system state. We demonstrate the potential to augment existing state-of-the-art autoscaling controllers with windowing and weighting methods to make more robust decisions, successfully mitigating over 90% of undesirable extreme parallelism shifts and significantly reducing scaling behaviour volatility.


Publication metadata

Author(s): Jamieson S, Forshaw M

Publication type: Conference Proceedings (inc. Abstract)

Publication status: Published

Conference Name: 17th ACM International Conference on Distributed and Event-based Systems (DEBS '23)

Year of Conference: 2023

Pages: 68-79

Print publication date: 10/06/2023

Online publication date: 27/06/2023

Acceptance date: 30/04/2023

Date deposited: 10/08/2023

Publisher: ACM

URL: https://doi.org/10.1145/3583678.3596886

DOI: 10.1145/3583678.3596886

Library holdings: Search Newcastle University Library for this item

ISBN: 9798400701221


Share