Processing of Hadoop using Highly Available NameNode
- Alexandra Angela Waters
- 2 years ago
- Views:
Transcription
1 College of Engineering Pune, India Abstract Hadoop is open source programming framework which enables the processing of large amount of data in distributed computing environment. Instead of depending on expensive hardware and different system for storing and processing data, Hadoop enables parallel processing on Big Data. They need to go through terabytes and petabytes of data to figure out particular queries and demands of users. Existing tools were becoming inadequate to process such large data sets. Hadoop solved the problem of handling such a huge amount of data. Hadoop has two core components:1. Hadoop Distributed File System (HDFS): It understands and assigns work to the nodes in a cluster.2. MapReduce: It is software framework used for distributed computing and processing of large amounts of data in the Hadoop cluster.in past few years, Hadoop Distributed File System (HDFS) has been used by many organizations with gigantic data sets and streams of operations on it. HDFS provides distinct features like, high fault tolerance, scalability, etc. The NameNode machine is a single point of failure (SPOF) for a HDFS cluster. If the NameNode machine fails, the system needs to be restarted manually, making the system less available. This paper proposes a highly available architecture and its working principle for the HDFS NameNode against its SPOF. Keywords HDFS, MapReduce, NameNode, SPOF I. INTRODUCTION There are so many organizations such as Facebook, Google, Yahoo, etc. in the world that owns very large amount of data on the internet. The main challenge in front of such organizations is to store and analyze huge quantities of data. To store and process such large amount of data, require special hardware machines, which is not a good deal in terms of money, as well as the processing capability of single machine cannot scale to process such huge data on it. Every single day huge amount of data is going to add from various fields like science, economics, engineering and commerce. To analyze such huge amounts of data for better understanding of users needs have led to development of data intensive applications and storage clusters. The basic requirement of these applications is a highly available, highly scalable and reliable cluster based storage system. Hadoop is an open source frame work used for processing such very large data sets. Hadoop develops software framework for reliable and scalable distributed computing. Hadoop mainly do this job with the help of Hadoop Distributed file system (HDFS) and MapReduce to perform computation on a cluster of commodity computers. II. EXISTING SYSTEM HDFS is doing its part of work efficiently in Hadoop since it was developed. HDFS works with two types of hardware machines, the (Slave machine) which is the machine on which application s data is stored and the NameNode (Master machine/server machine) which store the metadata of file system. Application s data is stored on using small blocks of data where 528
2 NameNode is the only single machine for storing metadata of file system and is the Single Point of Failure (SPOF) for the HDFS. SPOF of NameNode machine affects the overall availability of Hadoop. Hadoop provides high bandwidth to application data and suits with applications having large data sets. The core idea is to move large set of applications closer to where the data is located and perform computations on it. HDFS can scale up to ten thousands of low cost hardware machines on which data can be stored in the blocks and computation can be partition on these machines using MapReduce. MapReduce is a programming model used for processing very large data sets stored on HDFS clusters. NameNode is central point of focus in HDFS architecture as it is the only server machine which store metadata of file system; it manages namespace and grant access to file on the request of client where the is machine which stores the application s data. The data on is stored in small blocks which are made up by splitting large files. A. Namenode NameNode is the master of HDFS that directs the slave s to perform the low-level I/O tasks. NameNode keeps track of how files are broken down into file blocks, which nodes store those blocks, and the overall health of the distributed file system. The function of the NameNode is memory and I/O intensive. As such, the server hosting NameNode typically doesn t store any user data or perform any computations for a MapReduce program to lower the workload on the machine. This means that the NameNode server doesn t double as a or a. B. Each slave machine in a cluster will host a to perform the grunt work of the distributed file system - reading and writing HDFS blocks to actual files on the local file system. The data nodes can communicate to each other to rebalance data, move and copy data around and keep the replication high. Client can communicate directly with the to process the local files corresponding to the blocks. Metadata ops Read Datanodes Rack 2 Client Name Node Datanodes Rack 1 Replication Client Metadata (Name, Replicas,..) Fig 1.1 HDFS Architecture Block ops C. Hadoop Distributed File System (HDFS) Hadoop Distributed File System (HDFS) is a distributed file system designed to run on commodity hardware. HDFS has a high degree of fault-tolerance and is usually developed for deploying on low-cost hardware. As shown in the Fig 1.1, HDFS has master-slave architecture, with a single master called the NameNode and multiple slaves called s. NameNode manages and store the metadata of the file system. The metadata is maintained in the main memory of the NameNode to ensure fast access to the client, on read/write requests. s store and service read/write requests on files in HDFS, as directed by the NameNode. The files stored into HDFS are replicated into any number of s as per configuration, to ensure reliability and data availability. These replicas are distributed across the cluster to ensure rapid computations. HDFS provides efficient access to data and is appropriate for applications having big data sets
3 Files in HDFS are divided into smaller blocks, typically block size of 64MB, and each block is replicated and stored in multiple s. NameNode maintains the metadata for each file stored into HDFS, in its main memory. This includes a mapping between stored file names, the corresponding blocks of each file and the s that host these blocks. Hence, every request by a client to create, write, read or delete a file passes through the NameNode. Using the metadata stored, NameNode has to direct every request from client to the appropriate set of s. The client then communicates directly with the s to perform file operations. D. MapReduce MapReduce is a programming model and is opensource implementation of Hadoop, where MapReduce programs comprise of Map phase and Reduce phase - each of which defines a mapping from one set of key-value pairs to another. MapReduce programs can be written in various languages; Java, Ruby, Python, and C++. MapReduce programs are parallel, with Map tasks running concurrently, followed by their corresponding Reduce tasks that operate on data stored on the HDFS file System. Decomposing a data processing application into mappers and reducers is sometimes not important. But, once you write an application in the MapReduce form, scaling the application to run over hundreds, thousands of machines in a cluster is merely a configuration change. This specific scalability has attracted many programmers to use the MapReduce model. Hadoop MapReduce Framework comprises of two types of components that control the job execution process: a central component called the Job and a number of distributed components called s. Based on the location of a job s input data, the Job schedules tasks to run on some s and coordinates their execution of the job. s run tasks assigned to them and send progress reports to the Jobtracker. III. Secondary NameNode Name Node Job Fig 1.2 HDFS Architecture AVAILABILITY OF HDFS ARCHITECTURE The architecture of HDFS is designed to deal with frequent hardware failure as the core theme of Hadoop and cluster computing is that they run over lost cost commodity hardware machines, which are more susceptible to failure. Hence, the architecture of Hadoop was designed to deal with these failures. The other important feature of Hadoop is its scalability. It can scale up to ten thousands of machines. If you are using single hardware machine with some probability of failure, and on other side you are using ten thousands of such machines then probability of hardware failure will be ten thousand times great than that single machine. Hadoop was designed to have features like reliability, availability, fault tolerance, affordability and scalability. Hadoop exhibits all of these features very well except the availability feature that it has compromised a bit. The availability of Hadoop is very good when it comes to ; as blocks of data are being replicated at three different s. But when it comes to NameNode; Hadoop s availability is always on stack. If the NameNode machine goes down then the whole system becomes offline, as NameNode is the only entity which deals with the client queries and respond accordingly. It also keeps the precious metadata of file system. Hence with single NameNode machine the availability of Hadoop is always on stack, because it can go down at any time
4 IV. NAMENODE S SINGLE POINT OF FAILURE. Large Hadoop clusters have thousands of data nodes and one name node. The probability of failure goes up linearly with machine count. So if Hadoop didn't cope with data node failures it wouldn't scale. Since there's still only one name node, the Single Point of Failure (SPOF) is there. To eliminate the problem of HDFS NameNode, we have devised architecture of HDFS and its working principle. This architecture and working principle eliminates SPOF of NameNode by initially replicating NameNode at separate machines (in our case two machines) only once and then starts working normally with its s. The core idea of this architecture is that once NameNode get replicated after that, it will do all operation on its metadata by using well known 2- Phase Commit (2PC) Protocol. Initially the NameNode will be acting as Coordinator NameNode, and its replicas will be Participants NameNode machines. The Coordinator and Participants machines will perform any update to their metadata according to 2PC protocol. Due to 2PC protocol both of the replicas of Coordinator NameNode will be fully updated all time, hence in case Coordinator NameNode fails or even completely goes down then any of its replicas can become Coordinator and hosts the s. The new Coordinator can be elected using Election by Bully algorithm. V. PROPOSED SYSTEM To eliminate the Single Point of Failure (SPOF) of Hadoop HDFS Name Node, we have devised architecture of Name Node and its working principle under which the problem of Name Node SPOF is eliminated and the availability of Name Node is ensured. To eliminate the Single Point of Failure (SPOF) of Hadoop HDFS Name Node, we have devised architecture of Name Node and its working principle under which the problem of Name Node SPOF is eliminated and the availability of Name Node is ensured. A. Name Node Replication To increase the availability of any system, the very simple technique that can be used is redundancy. Here at initial step, we have done the same thing by replicating Name Node at two different machines that are called Participant Name Nodes. In our case we have taken replica factor two, but it can be varied according to degree of availability requirement. Hence when the Name Node starts for the first time, it will be replicated only once on Participant Name Node machines. Once replication is complete, the system will then work under the principle of Two Phase Commit Protocol with some modifications. Before discussing our proposed modified form of 2PC protocol, the actual 2PC protocol is discussed below. B. Two Phase NameNode Commit (2PNNC) Protocol In our proposed solution once NameNode is replicated, after that NameNode and its replicas will work under the principle of Two Phase Commit (2PC) Protocol. We have not exactly used the same 2PC protocol, but have made some variations in it according to our needs. Here the NameNode will be called Coordinator NameNode and its replicas will be known as Participant NameNode machines. Every Participant NameNode is associated with an update-queue that will hold multiple vote-requests from the Coordinator. Any of the Participants can send a votecommit to coordinator, as soon as it receives the vote-request and its update-queue is empty. So, in 531
5 this case the Coordinator need not wait for all of the vote-commit replies from the Participants, rather it will continue to send global-commit against a single vote-commit from any of the Participant. The remaining Participants can perform their commit operation later, but there must be at least one Participant who will update itself with the Coordinator at same time. After the vote-request from the Coordinator, if the update-queue is partially filled then the Participant can accept the voterequests, but will send vote-commit to coordinator until it has performed all the pending operations in its update-queue. The Participant will send voteabort in the case when its update-queue is full. The 2PNNC protocol for our proposed solution is explained below: Phase 1a. Coordinator NameNode sends vote request to all Participant NameNodes. Phase 1b. When any of the Participant NameNode receives vote-request from Coordinator then it will either reply vote-commit or vote-abort. It will send vote-abort if its update-queue is full of previous voterequests, otherwise it will keep the vote-request in its update-queue and sends vote-commit to coordinator when its update-queue become empty by performing all previous requested updates. Phase 2a. The Coordinator NameNode collects all votes from the Participants. If it gets vote-commit from any one of its Participants NameNode, it commits its update operation and sends out globalcommit to all Participant NameNodes. If it receives vote-abort from any of its Participants NameNodes then it will abort its update operation and will send global-abort to all Participant NameNodes. Phase 2b. Every Participants NameNode machines will wait for Coordinator global-commit and globalabort and will respond accordingly. Vote Request Vote Abort Global Abort Commit ACK Abort State Vote Request Vote Commit Global ACK Fig 1.3 Participant NameNode Working Principle under 2PNNC protocol VI. CONCLUSION In this paper we have proposed the implementation of Hadoop to deal with SPOF problem of NameNode machine. This will greatly increase the availability of Hadoop by eliminating NameNode s SPOF. We have utilized two wellknown techniques of distributed systems, the 2PC protocol and Election by Bully algorithm, with some variations in these techniques by reducing their waiting time and increasing their throughput. Both of these techniques are widely deployed and adopted successfully. VII. Initial State Ready State Commit State REFERENCES [1] Prashant D. Londhe, Satish S. Kumbhar, Ramakant S. Sul, Amit J. Khadse, PROCESSING BIG DATA USING HADOOP FRAMEWORK, Proceedings of 4th SARC- IRF International Conference, New Delhi, India, April. 2014, ISBN: [2] Mohammad Asif Khan, Zulfiqar A. Memon, Sajid Khan, Highly Available Hadoop NameNode Architecture, 2012 International Conference on Advanced Computer Science Applications and Technologies. [3] Dhruba Borthakur, The Hadoop Distributed File System:Architecture and Design, in Apache Software foundation, hadoop.apache.org/common/docs /r0.18.0/hdfs _design.pdf
6 [4] Konstantin Shvachko, Hairong Kuang, Sanjay Radia, Robert Chansler Yahoo!, The Hadoop Distributed File System, IEEE NASA storage conference, storageconference.org/2010/papers/msst/ Shvachko.pdf. [5] Aaron Myers, High Availability for the Hadoop Distributed File System (HDFS), Article at Cloudera, March 07, [6] Tom White, Hadoop: The Definitive Guide. O Reilly Published by O Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472,
Hadoop Distributed File System. Jordan Prosch, Matt Kipps
Hadoop Distributed File System Jordan Prosch, Matt Kipps Outline - Background - Architecture - Comments & Suggestions Background What is HDFS? Part of Apache Hadoop - distributed storage What is Hadoop?
INTERNATIONAL JOURNAL OF PURE AND APPLIED RESEARCH IN ENGINEERING AND TECHNOLOGY
INTERNATIONAL JOURNAL OF PURE AND APPLIED RESEARCH IN ENGINEERING AND TECHNOLOGY A PATH FOR HORIZING YOUR INNOVATIVE WORK A COMPREHENSIVE VIEW OF HADOOP ER. AMRINDER KAUR Assistant Professor, Department Distributed File System (HDFS) Overview
2012 coreservlets.com and Dima May Hadoop Distributed File System (HDFS) Overview Originals of slides and source code for examples: Also see the customized
Distributed File Systems
Distributed File Systems Mauro Fruet University of Trento - Italy 2011/12/19 Mauro Fruet (UniTN) Distributed File Systems 2011/12/19 1 / 39 Outline 1 Distributed File Systems 2 The Google File System (GFS)
CDH AND BUSINESS CONTINUITY:
WHITE PAPER CDH AND BUSINESS CONTINUITY: An overview of the availability, data protection and disaster recovery features in Hadoop Abstract Using the sophisticated built-in capabilities of CDH for tun
Hadoop and Map-Reduce. Swati Gore
Hadoop and Map-Reduce Swati Gore Contents Why Hadoop? Hadoop Overview Hadoop Architecture Working Description Fault Tolerance Limitations Why Map-Reduce not MPI Distributed sort Why Hadoop? Existing Data
The Hadoop distributed file system
The Hadoop distributed file system Pooja S.Honnutagi Computer Science & Engineering VTU Regional Centre Gulbaga,Karnataka,India Abstract: The flood of data generated from many sources daily. Paul Krzyzanowski Rutgers University October 28, 2012 1 Introduction The classic network file systems we examined, NFS, CIFS, AFS, Coda, were designed as client-server applications. & MapReduce Hadoop is an open-source software framework (or platform) for Reliable + Scalable + Distributed Storage/Computational unit Failures completely
!"#$%&' ( )%#*'+,'-#.//"0( !"#$"%&'()*$+()',!-+.'/', 4(5,67,!-+!"89,:*$;'0+$.<.,&0$'09,&)"/=+,!()<>'0, 3, Processing LARGE data sets
!"#$%&' ( Processing LARGE data sets )%#*'+,'-#.//"0( Framework for o! reliable o! scalable o! distributed computation of large data sets 4(5,67,!-+!"89,:*$;'0+$.,
Volume 3, Issue 6, June 2015 International Journal of Advance Research in Computer Science and Management Studies
Volume 3, Issue 6, June 2015 International Journal of Advance Research in Computer Science and Management Studies Research Article / Survey Paper / Case Study Available online at: Image
Large scale processing using Hadoop. Ján Vaňo
Large scale processing using Hadoop Ján Vaňo What is Hadoop? Software platform that lets one easily write and run applications that process vast amounts of data Includes: MapReduce offline computing engine Distributed File System: Architecture and Design
Hadoop Distributed File System: Architecture and Design Gayatri G. Divekar 1, Liza R. Daiya 2 and Prof. Vivek. R. Shelke 3 1Computer Science & Engineering Jawaharlal Darda Institute of Engineering Parallel Data Processing
MapReduce and Implementation Hadoop Parallel Data Processing Kai Shen A programming interface (two stage Map and Reduce) and system support such that: the interface is easy to program, and suitable for
Contents. 1. Introduction
Summary Cloud computing has become one of the key words in the IT industry. The cloud represents the internet or an infrastructure for the communication between all components, providing and receiving
A REVIEW: Distributed File System
Journal of Computer Networks and Communications Security VOL. 3, NO. 5, MAY 2015, 229 234 Available online at: EISSN 23089830 (Online) / ISSN 24100595 (Print) A REVIEW: System Shiva Asadianfam,
Application Development. A Paradigm Shift
Application Development for the Cloud: A Paradigm Shift Ramesh Rangachar Intelsat t 2012 by Intelsat. t Published by The Aerospace Corporation with permission. New 2007 Template - 1 Motivation for the