Asynchronous Non-Blocking Algorithm to Handle Straggler Reduce Tasks in Hadoop System

Khoiruddin, A.A. and Zakaria, N. and Alhussian, H. (2020) Asynchronous Non-Blocking Algorithm to Handle Straggler Reduce Tasks in Hadoop System. International Journal on Advanced Science, Engineering and Information Technology, 10 (5). pp. 1913-1919. ISSN 20885334

Full text not available from this repository.
Official URL: https://www.scopus.com/inward/record.uri?eid=2-s2....

Abstract

Hadoop is widely adopted as a big data processing application as it can run on commercial hardware at a reasonable time. Hadoop uses asynchronous blocking concurrency using Thread and Future class. Therefore, in some cases such as network link or hardware failure, a running task may block other tasks from running (the task becomes straggler). Hadoop releases are equipped with algorithms to handle straggler tasks problem. However, the algorithms manage Map and Reduce task similarly, while the straggler root cause might be different for both tasks. In this paper, the Asynchronous Non-Blocking (ANB) method is proposed to improve the performance and avoid the blocking of Reduce task in Hadoop. Instead of using the single queue, our approach uses two queues, i.e. task queue and callback queue. When a task is not ready or detected as a straggler, it is removed from the main task queue and temporarily sent to the callback queue. When the task is ready to run, it will be sent back to the main task queue for running. The performance of the algorithm is compared with rTuner, the latest paper found on handling straggler task in Reduce task. From the comparison, it is shown that ANB consistently gives faster time to complete because any unready tasks will be directly put into the callback queue without blocking other tasks. Furthermore, the overhead time in rTuner is high as it needs to check the straggler status and to find the reason for a task to become straggler. ©

Item Type: Article
Additional Information: cited By 0
Depositing User: Mr Ahmad Suhairi UTP
Date Deposited: 10 Nov 2023 03:28
Last Modified: 10 Nov 2023 03:28
URI: https://khub.utp.edu.my/scholars/id/eprint/13682

Actions (login required)

View Item
View Item