Login Register Cart 0
Generic placeholder image

Recent Advances in Computer Science and Communications

Editor-in-Chief

ISSN (Print): 2666-2558
ISSN (Online): 2666-2566

Back
Journal Home About Journal Editorial Board Current Issue Volumes/Issues
Subscribe
Research Article

Efficient Dynamic Resource Allocation in Hadoop Multiclusters for Load- Balancing Problem

Author(s): Karthikeyan S.*, Hari Seetha and Manimegalai R.

Volume 13, Issue 4, 2020

Page: [686 - 693] Pages: 8

DOI: 10.2174/2213275912666190430161947

Price: $65

Abstract

Background: ‘Map-Reduce’ is the framework and its processing of data by rationalizing the distributed servers. also its running the various tasks in parallel way. The most important problem in map reduce environment is Resource Allocation in distributed environments and data locality to its corresponding slave nodes. If the applications are not scheduled properly then it leads to load unbalancing problems in the cloud environments.

Objective: This Research suggests a new dynamic way of allocating the resources in hadoop multi cluster environment in order to effectively monitor the nodes for faster computation, load balancing and data locality issues. The dynamic slot allocation is explained theoretically in order to address the problems of pre configuration, speculative execution, delay in scheduling and pre slot allocation in hadoop environments. Experiment is done with Hadoop cluster which increases the efficiency of the nodes and solves the load balancing problem.

Methods: The Current design of Map Reduce Hadoop systems are affected by underutilization of slots. The reason is the number of maps and reducer is allotted is smaller than the available number of maps and reducers. In Hadoop most of the times its noticed that dynamic slot allocation policy, the mapper or reducers are idle. So we can use those unused map tasks to overloaded reducer tasks in-order to increase the efficiency of MR jobs and vice versa.

Results: The real time experiment was implemented with Multinode Hadoop cluster map reduce jobs of file size 1giga bytes to 5 giga bytes and various performance test has been taken.

Conclusion: This paper focused on Hadoop map reduce resource allocation management techniques for multi cluster environments. It proposes a novel dynamic slot allocation policy to improve the performance of yarn scheduler and eliminates the load balancing problem. This work proves that dynamic slot allocation is outperforms more than yarn framework. In future it considered to concentrate on CPU bandwidth and processing time.

Keywords: Cloud-hadoop-cluster, mapreduce, resource allocation-yarn-dynamic, yarn framework, hadoop map, speculative execution.

Graphical Abstract

[1]
M.M. Kumar, N.R. Vikram, S.S. Sakthi Kavin, and K. Sathya, "A novel chicken pecking order algorithm for efficient map-reduce", 2018 Second International Conference on Inventive Communication and Computational Technologies (ICICCT) Coimbatore, 2018pp. 1685-1689
[http://dx.doi.org/10.1109/ICICCT.2018.8473147]
[2]
A.S. Lakshmi, M.B. Raju, and N.S. Chandra, "Towards optimization of HadoopMapReduce jobs on cloud 2016", International Conference on Computing, Analytics and Security Trends (CAST), Pune, 2016, pp. 255-260,
[http://dx.doi.org/ 10.1109/CAST. 2016.7914976]
[3]
L Y. Ho, and P. Liu, Optimal algorithms for cross rack communication optimization in MapReduce framework,”; IEEE Cloud,. July 2011, pp.420-427,
[4]
Y. Yao, B. Sheng, and C.C. Tan, " “Self adjusting slot configurations for homogeneous and heterogeneous Hadoop cluster”,", IEEE transactions in Cloud Computing. March 2015, pp. 344-357,
[5]
X. Dong, H. Liao, and Y. Wang, "“Scheduling real-time mixed time and non-real-time applications in map-reduce environments” In ", 2011 IEEE 17th International Conference on Parallel and Distributed Systems 2011, pp. 9-16.,
[6]
R.N. Calheirs, and M. Mattess, "“Scaling map-reduce applications across hybrid clouds to meet soft deadlines”", 2013 IEEE 27th International Conference on Advanced Information Networking and Applications (AINA).
[7]
Q. Zhang, M. F. Zhani, Y. Yang, R. Boutaba, and B. Wong, ""PRISM: Fine-Grained resource-aware scheduling for mapreduce,"", IEEE Trans. Cloud Comput.. Vol. 3, no. 2, pp. 182-194, 2015. ,
[http://dx.doi.org/10.1109/TCC.2014. 2379096]
[8]
X. Wang, X. Wang, H. Che, K. Li, M. Huang, and C. Gao, "An intelligent economic approach for dynamic resource allocation in cloud services", IEEE Trans. Cloud Comput., vol. 3, no. 3, pp. 275-289, 2015.
[http://dx.doi.org/10.1109/TCC.2015.2415776]
[9]
Y. Wei, L. Pan, S. Liu, L. Wu, and X. Meng, "DRL-Scheduling: An intelligent qos-aware job scheduling framework for applications in clouds", IEEE Access, vol. 6, pp. 55112-55125, 2018.
[http://dx.doi.org/10.1109/ACCESS.2018.2872674]
[10]
S. Tang, B. Lee, and B. He, " "DynamicMR: A dynamic slot allocation optimization framework for mapreduce clusters," ", IEEE Trans. Cloud Comput..vol. 2, no. 3, pp. 333-347, 2014.,
[http://dx.doi.org/10.1109/TCC.2014.2329299]
[11]
J. Sahni, and D.P. Vidyarthi, "A cost-effective deadline-constrained dynamic scheduling algorithm for scientific workflows in a cloud environment", IEEE Trans. Cloud Comput., vol. 6, no. 1, pp. 2-18, 2018.
[http://dx.doi.org/10.1109/TCC.2015.2451649]
[12]
L. Shi, Z. Zhang, and T. Robertazzi, "Energy-aware scheduling of embarrassingly parallel jobs and resource allocation in cloud", IEEE Trans. Parallel Distrib. Syst., vol. 28, no. 6, pp. 1607-1620, 2017.
[http://dx.doi.org/10.1109/TPDS.2016.2625254]
[13]
N. Lim, S. Majumdar, and P. Ashwood-Smith, "MRCP-RM: A technique for resource allocation and scheduling of mapreduce jobs with deadlines", IEEE Transact Parall Distrib. Syst., vol. 28, no. 5, pp. 1375-1389, 2017.
[14]
S.H. Adil, K. Raza, U. Ahmed, S.S.A. Ali, and M. Hashmani, "Cloud task scheduling using nature inspired meta-heuristic algorithm", 2015 International Conference on Open Source Systems & Technologies (ICOSST, 2015pp. 158-164 Lahore,
[http://dx.doi.org/10.1109/ICOSST.2015.7396420]
[15]
M. A. Rodriguez, and R. Buyya, "Deadline based resource provisioning and scheduling algorithm for scientific workflows on clouds", IEEE Trans. Cloud Comput., Vol. 2, no. 2, pp. 222-235, 2014.
[http://dx.doi.org/10.1109/TCC.2014. 2314655]

Rights & Permissions Print Cite
Article Metrics
9
© 2024 Bentham Science Publishers | Privacy Policy