ePrints

Pluggable AI-based real-time stragglers detection framework in Hadoop

Lookup NU author(s): Xinyang Liu, Yinhao Li, Professor Raj Ranjan, Dr Dev JhaORCiD

Downloads

• Published version [.pdf]

Licence

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License (CC BY-NC-ND).

Abstract

© 2025 The Author(s).The growing reliance on big data frameworks such as Hadoop has revolutionized data processing across various domains, enabling large-scale storage and distributed computation. Hadoop is widely employed in real-world applications such as high-performance computation tasks, e-commerce and data analysis in healthcare. However, the efficiency of Hadoop systems is often hampered by faults and anomalies, with stragglers emerging as one of the most prevalent issues. Stragglers disrupt workflows, waste resources and degrade system performance. While existing anomaly detection models employ methods like median analysis or static thresholds, they often struggle with issues such as high false positives, lack of adaptability and poor handling of complex heterogeneous environments. To address these challenges, this paper presents Plabs , a flexible stragglers detection framework for Hadoop. The framework comprises two core components: (1) a Monitoring Module providing real-time tracking of cluster resources and task progress and (2) a Pluggable AI-based straggler detection module, designed for precise straggler task identification. By leveraging advanced monitoring and AI-driven analysis, Plabs offers an automated, flexible and scalable solution for detecting stragglers at run-time in Hadoop clusters. We evaluated Plabs exhaustively with three Machine Learning (ML), two Deep Learning (DL) and two Large Language Models (LLMs) on five different applications in a real testbed environment. Our experiment evaluation shows that DL models outperform others in identifying Hadoop stragglers, achieving superior accuracy and reliability for all the applications.

Publication metadata

Author(s): Liu X, Li Y, Ranjan R, Jha DN

Publication type: Article

Publication status: Published

Journal: High-Confidence Computing

Year: 2026

Volume: 6

Issue: 1

Print publication date: 01/03/2026

Online publication date: 03/07/2025

Acceptance date: 20/06/2025

Date deposited: 02/02/2026

ISSN (electronic): 2667-2952

Publisher: Shandong University

URL: https://doi.org/10.1016/j.hcc.2025.100341

DOI: 10.1016/j.hcc.2025.100341

Altmetrics

Funding

Share