Capacity-aware key partitioning scheme for heterogeneous big data analytic engines
DC Field | Value | Language |
---|---|---|
dc.contributor.author | Hanif, Muhammad | - |
dc.contributor.author | Lee, Choonhwa | - |
dc.date.accessioned | 2022-07-12T07:37:33Z | - |
dc.date.available | 2022-07-12T07:37:33Z | - |
dc.date.created | 2021-05-13 | - |
dc.date.issued | 2018-03 | - |
dc.identifier.issn | 1738-9445 | - |
dc.identifier.uri | https://scholarworks.bwise.kr/hanyang/handle/2021.sw.hanyang/150409 | - |
dc.description.abstract | Big data and cloud computing became the centre of interest for the past decade. With the increase of data size and different cloud application, the idea of big data analytics become very popular both in industry and academia. The research communities in industry and academia never stopped trying to come up with the fast, robust, and fault tolerant analytic engines. MapReduce becomes one of the popular big data analytic engine over the past few years. Hadoop is a standard implementation of MapReduce framework for running data-intensive applications on the clusters of commodity servers. By thoroughly studying the framework we find out that the shuffle phase, all-to-all input data fetching phase in reduce task significantly affect the application performance. There is a problem of variance in both the intermediate key's frequencies and their distribution among data nodes throughout the cluster in Hadoop's MapReduce system. This variance in system causes network overhead which leads to unfairness on the reduce input among different data nodes in the cluster. Because of the above problems, applications experience performance degradation due to shuffle phase of MapReduce applications. We develop a new novel algorithm; unlike previous systems our algorithm considers each node's capabilities as heuristics to decide a better available trade-off for the locality and fairness in the system. By comparing with the default Hadoop's partitioning algorithm and Leen partitioning algorithm: a). In case of 2 million key-value pairs to process, on the average our approach achieve better resource utilization by about 19%, and 9%, in that order; b). In case of 3 million key-value pairs to process, our approach achieve near optimal resource utilization by about 15%, and 7%, respectively. | - |
dc.language | 영어 | - |
dc.language.iso | en | - |
dc.publisher | Institute of Electrical and Electronics Engineers Inc. | - |
dc.title | Capacity-aware key partitioning scheme for heterogeneous big data analytic engines | - |
dc.type | Article | - |
dc.contributor.affiliatedAuthor | Lee, Choonhwa | - |
dc.identifier.doi | 10.23919/ICACT.2018.8323922 | - |
dc.identifier.scopusid | 2-s2.0-85046772696 | - |
dc.identifier.bibliographicCitation | International Conference on Advanced Communication Technology, ICACT, v.2018-February, pp.999 - 1007 | - |
dc.relation.isPartOf | International Conference on Advanced Communication Technology, ICACT | - |
dc.citation.title | International Conference on Advanced Communication Technology, ICACT | - |
dc.citation.volume | 2018-February | - |
dc.citation.startPage | 999 | - |
dc.citation.endPage | 1007 | - |
dc.type.rims | ART | - |
dc.type.docType | Conference Paper | - |
dc.description.journalClass | 1 | - |
dc.description.isOpenAccess | N | - |
dc.description.journalRegisteredClass | scopus | - |
dc.subject.keywordPlus | Axial flow | - |
dc.subject.keywordPlus | Economic and social effects | - |
dc.subject.keywordPlus | Electronic trading | - |
dc.subject.keywordPlus | Engines | - |
dc.subject.keywordPlus | Multiprocessing systems | - |
dc.subject.keywordPlus | Application performance | - |
dc.subject.keywordPlus | Context-Aware | - |
dc.subject.keywordPlus | Data-intensive application | - |
dc.subject.keywordPlus | Hadoop MapReduce | - |
dc.subject.keywordPlus | Heterogeneous systems | - |
dc.subject.keywordPlus | Partitioning algorithms | - |
dc.subject.keywordPlus | Performance degradation | - |
dc.subject.keywordPlus | Resource utilizations | - |
dc.subject.keywordPlus | Big data | - |
dc.subject.keywordAuthor | Cloud and Distributed Computing | - |
dc.subject.keywordAuthor | Context-aware Partitioning | - |
dc.subject.keywordAuthor | Hadoop MapReduce | - |
dc.subject.keywordAuthor | Heterogeneous Systems | - |
dc.identifier.url | https://ieeexplore.ieee.org/document/8323922 | - |
Items in ScholarWorks are protected by copyright, with all rights reserved, unless otherwise indicated.
222, Wangsimni-ro, Seongdong-gu, Seoul, 04763, Korea+82-2-2220-1365
COPYRIGHT © 2021 HANYANG UNIVERSITY.
Certain data included herein are derived from the © Web of Science of Clarivate Analytics. All rights reserved.
You may not copy or re-distribute this material in whole or in part without the prior written consent of Clarivate Analytics.