Abstract
Background: ‘Map-Reduce’ is the framework and its processing of data by rationalizing the distributed servers. also its running the various tasks in parallel way. The most important problem in map reduce environment is Resource Allocation in distributed environments and data locality to its corresponding slave nodes. If the applications are not scheduled properly then it leads to load unbalancing problems in the cloud environments.
Objective: This Research suggests a new dynamic way of allocating the resources in hadoop multi cluster environment in order to effectively monitor the nodes for faster computation, load balancing and data locality issues. The dynamic slot allocation is explained theoretically in order to address the problems of pre configuration, speculative execution, delay in scheduling and pre slot allocation in hadoop environments. Experiment is done with Hadoop cluster which increases the efficiency of the nodes and solves the load balancing problem.
Methods: The Current design of Map Reduce Hadoop systems are affected by underutilization of slots. The reason is the number of maps and reducer is allotted is smaller than the available number of maps and reducers. In Hadoop most of the times its noticed that dynamic slot allocation policy, the mapper or reducers are idle. So we can use those unused map tasks to overloaded reducer tasks in-order to increase the efficiency of MR jobs and vice versa.
Results: The real time experiment was implemented with Multinode Hadoop cluster map reduce jobs of file size 1giga bytes to 5 giga bytes and various performance test has been taken.
Conclusion: This paper focused on Hadoop map reduce resource allocation management techniques for multi cluster environments. It proposes a novel dynamic slot allocation policy to improve the performance of yarn scheduler and eliminates the load balancing problem. This work proves that dynamic slot allocation is outperforms more than yarn framework. In future it considered to concentrate on CPU bandwidth and processing time.
Keywords: Cloud-hadoop-cluster, mapreduce, resource allocation-yarn-dynamic, yarn framework, hadoop map, speculative execution.
Graphical Abstract
[http://dx.doi.org/10.1109/ICICCT.2018.8473147]
[http://dx.doi.org/ 10.1109/CAST. 2016.7914976]
[http://dx.doi.org/10.1109/TCC.2014. 2379096]
[http://dx.doi.org/10.1109/TCC.2015.2415776]
[http://dx.doi.org/10.1109/ACCESS.2018.2872674]
[http://dx.doi.org/10.1109/TCC.2014.2329299]
[http://dx.doi.org/10.1109/TCC.2015.2451649]
[http://dx.doi.org/10.1109/TPDS.2016.2625254]
[http://dx.doi.org/10.1109/ICOSST.2015.7396420]
[http://dx.doi.org/10.1109/TCC.2014. 2314655]
Related Journals
International Journal of Sensors, Wireless Communications and Control
Related Books
Challenges and Opportunities for Deep Learning Applications in Industry 4.0
Recent Advances in IoT and Blockchain Technology
Augmented Intelligence: Deep Learning, Machine Learning, Cognitive Computing, Educational Data Mining
Human-Computer Interaction and Beyond: Advances Towards Smart and Interconnected Environments Part II
Machine Learning and Its Application: A Quick Guide for Beginners
Computational Intelligence For Data Analysis
Applied Machine Learning and Multi-Criteria Decision-Making in Healthcare
Advanced Computing Techniques: Implementation, Informatics and Emerging Technologies
How to Design Optimization Algorithms by Applying Natural Behavioral Patterns
Human-Computer Interaction and Beyond: Advances Towards Smart and Interconnected Environments (Part I)
9