Manchester Metropolitan University's Research Repository

    Dynamic Container-based Resource Management Framework of Spark Ecosystem

    Faseeh Qureshi, NM, Farah Siddiqui, I, Abbas, A, Bashir, AK ORCID logoORCID: https://orcid.org/0000-0001-7595-2522, Choi, K, Kim, J and Shin, DR (2019) Dynamic Container-based Resource Management Framework of Spark Ecosystem. In: 21st International Conference on Advanced Communication Technology (ICACT), 17 February 2019 - 20 February 2019, PyeongChang, South Korea.


    Download (4MB) | Preview


    © 2019 Global IT Research Institute (GIRI). Apache Spark is known for its robustness in processing large-scale datasets in a distributed computing environment. This form of efficiency is highly observing because of the direct use of Random-Access Memory (RAM) in processing its resilient distributed datasets across the ecosystem. Recently, it is observed that, the memory utilization in computing spark jobs is mainly dependent on job containers, which are closely associated to persistent storage media components. Thus, spark jobs processing relevancy is tightly coupled to the type of storage container and in case of any dynamic resource allocation, the job loses its ratio of resource computation in existing container and increases a functional issue of processing large-scale datasets in spark ecosystem. In this paper, we propose dynamic container-based resource management framework, that shifts coupled associations of job profiles to dynamically available resource containers. Also, it relieves static container allocations and presumes them as a fresh piece of resource allocation for new job profile. The experimental evaluation shows that the proposed dynamic framework reduces wastage of resource allocations and increase ecosystem performance than default job profile in spark ecosystem.

    Impact and Reach


    Activity Overview
    6 month trend
    6 month trend

    Additional statistics for this dataset are available via IRStats2.


    Repository staff only

    Edit record Edit record