Author : Mr. Jitesh J. Patil 1
Date of Publication :7th June 2016
Abstract: The Map Reduce framework and its open source implementation in Hadoop is existing as a standard for Bigdata related processing in industry and academies. When a bunch of jobs are simultaneously submitted together to a Map Reduce cluster, bunch of jobs will compete for available resources by this the overall system performance may go down, this is because in Map Reduce cluster different kinds of workload is shared among multiple users. Existing scheduling algorithms which are supported by Hadoop always cannot guarantee good average response time with different workloads. Therefore it is a challenging ability to design an effective scheduler which can work with shared Map Reduce cluster. To solve this problem we proposed a new hadoop scheduler which works on the different workload patterns and reduces overall job response time by using the knowledge of workload patterns.. The scheduler will reduce the average job response time that are compared with existing Fair and FIFO Scheduler.
Reference :
-
- Yi Yao, Jianzhe Tai, Bo Sheng, and Ningfang Mi, LsPS: A Job Size-Based Scheduler for Efficient Task Assignments in Hadoop, IEEE Transactions on Cloud Computing, no. 1, pp. 1, PrePrints PrePrints, doi:10.1109/TCC.2014.2338291.
- J. Dean, S. Ghemawat, and G. Inc., Map Reduce: Simplified data processing on large clusters, in Proc. 6th Conf. Symp. Operating Syst. Des. Implementation, 2004, p.10.
- M. Zaharia, D. Borthakur, J. S. Sarma, K. Elmeleegy, S. Shenker, and I. Stoica, Job scheduling for multi-user Map Reduce clusters, University of California, Berkeley, CA, USA, Tech. Rep. UCB/ EECS-2009-55 , Apr. 2009.
- Y. Chen, A. Ganapathi, R. Griffith, and R. Katz, The case for evaluating Map Reduce performance using workload suites, in Proc. IEEE 19th Int. Symp. Model., Anal. Simul. Comput. Telecommun. Syst., 2011, pp. 390399.
- https://hadoop.apache.org/docs/r1.2.1/fair_scheduler.html
- https://hadoop.apache.org/docs/r1.2.1/capacity_scheduler.ht ml