e-space
Manchester Metropolitan University's Research Repository

Dynamic Container-based Resource Management Framework of Spark Ecosystem

Faseeh Qureshi, NM and Farah Siddiqui, I and Abbas, A and Bashir, AK and Choi, K and Kim, J and Shin, DR (2019) Dynamic Container-based Resource Management Framework of Spark Ecosystem. In: 21st International Conference on Advanced Communication Technology (ICACT), 17 February 2019 - 20 February 2019, PyeongChang, South Korea.

[img]
Restricted to Repository staff only until 2 May 2021.

Download (4MB)

Abstract

© 2019 Global IT Research Institute (GIRI). Apache Spark is known for its robustness in processing large-scale datasets in a distributed computing environment. This form of efficiency is highly observing because of the direct use of Random-Access Memory (RAM) in processing its resilient distributed datasets across the ecosystem. Recently, it is observed that, the memory utilization in computing spark jobs is mainly dependent on job containers, which are closely associated to persistent storage media components. Thus, spark jobs processing relevancy is tightly coupled to the type of storage container and in case of any dynamic resource allocation, the job loses its ratio of resource computation in existing container and increases a functional issue of processing large-scale datasets in spark ecosystem. In this paper, we propose dynamic container-based resource management framework, that shifts coupled associations of job profiles to dynamically available resource containers. Also, it relieves static container allocations and presumes them as a fresh piece of resource allocation for new job profile. The experimental evaluation shows that the proposed dynamic framework reduces wastage of resource allocations and increase ecosystem performance than default job profile in spark ecosystem.

Impact and Reach

Statistics

Downloads
Activity Overview
1Download
33Hits

Additional statistics for this dataset are available via IRStats2.

Altmetric

Actions (login required)

Edit Item Edit Item