HDFS:

Hadoop Distributed
File System
CIS 612
Sunnie Chung
 Introduction
• What is Big Data??
– Bulk Amount
– Unstructured

• Lots of Applications which need to handle

huge amount of data (in terms of 500+ TB per
day)
• If a regular machine need to transmit 1TB of
data through 4 channels : 43 Minutes.
• What if 500 TB ??
Hadoop 2
 What is Hadoop?

• Framework for large-scale data processing

• Inspired by Google’s Architecture:
– Google File System (GFS) and MapReduce
• Open-source Apache project
– Nutch search engine project
– Apache Incubator
• Written in Java and shell scripts

Hadoop 3
Hadoop Distributed File System (HDFS)

• Storage unit of Hadoop

• Relies on principles of Distributed File System.
• HDFS have a Master-Slave architecture
• Main Components:
– Name Node : Master
– Data Node : Slave

• 3+ replicas for each block

• Default Block Size : 128MB

Hadoop 4
 H
Hadoop Distributed File System (HDFS)

• Hadoop Distributed File System (HDFS)

– Runs entirely in userspace
– The file system is dynamically distributed across multiple
computers
– Allows for nodes to be added or removed easily
– Highly scalable in a horizontal fashion

• Hadoop Development Platform

– Uses a MapReduce model for working with data
– Users can program in Java, C++, and other languages

Hadoop 5
 Why should I use Hadoop?

• Fault-tolerant hardware is expensive

• Hadoop designed to run on commodity hardware

• Automatically handles data replication and deals with

node failure

• Does all the hard work so you can focus on processing

data

Hadoop 6
 HDFS: Key Features
• Highly Fault Tolerant:
Automatic Failure Recovery System
• High aggregate throughput for streaming large files
• Supports replication and locality features
• Designed to work with systems with vary large file
(files with size in TB) and few in number.
• Provides streaming access to file system data. It is
specifically good for write once read many kind of
files (for example Log files).

Hadoop 7
Hadoop Distributed File System (HDFS)

• Can be built out of commodity hardware. HDFS

doesn't need highly expensive storage devices
– Uses off the shelf hardware
• Rapid Elasticity
– Need more capacity, just assign some more nodes
– Scalable
– Can add or remove nodes with little effort or
reconfiguration
• Resistant to Failure
• Individual node failure does not disrupt the
system
Hadoop 8
Who uses Hadoop?

Hadoop 9
 What features does Hadoop offer?

• API and implementation for working with

MapReduce
• Infrastructure
– Job configuration and efficient scheduling
– Web-based monitoring of cluster stats
– Handles failures in computation and data nodes
– Distributed File System optimized for huge amounts of
data

Hadoop 10
 When should you choose Hadoop?

• Need to process a lot of unstructured data

• Processing needs are easily run in parallel

• Batch jobs are acceptable

• Access to lots of cheap commodity machines

Hadoop 11
 When should you avoid Hadoop?

• Intense calculations with little or no data

• Processing cannot easily run in parallel

• Data is not self-contained

• Need interactive results

Hadoop 12
 Hadoop Examples

• Hadoop would be a good choice for:

– Indexing log files
– Sorting vast amounts of data
– Image analysis
– Search engine optimization
– Analytics
• Hadoop would be a poor choice for:
– Calculating Pi to 1,000,000 digits
– Calculating Fibonacci sequences
– A general RDBMS replacement

Hadoop 13
Hadoop Distributed File System (HDFS)

• How does Hadoop work?

– Runs on top of multiple commodity systems
– A Hadoop cluster is composed of nodes
• One Master Node
• Many Slave Nodes
– Multiple nodes are used for storing data & processing
data
– System abstracts the underlying hardware to
users/software

Hadoop 14
 How HDFS works: Split Data

• Data copied into HDFS is split into blocks

• Typical HDFS block size is 128 MB
– (VS 4 KB on UNIX File Systems)

Hadoop 15
 How HDFS works: Replication

• Each block is replicated to multiple machines

• This allows for node failure without data loss

Block Block Block

#1 #2 #1
Data Data Data
Node 1 Node 2 Node 3
Block Block Block
#2 #3 #3
Hadoop 16
HDFS Architecture
 Hadoop Distributed File System (HDFS)p:

HDFS
HDFS is a multi-node system • HDFS Consists of data blocks
– Files are divided into data

Name Node (Master)
blocks

Single point of failure
– Default size if 64MB

Data Node (Slave)
– Default replication of blocks is 3

Failure tolerant (Data
– Blocks are spread out over Data
replication) Nodes

Hadoop 18
 Hadoop Architecture Overview

Client

Job Tracker

Task Tracker Task Tracker

Name
Node
Data Data
Node Data Node
Data
Node Node

Hadoop 19
 Hadoop Components: Job Tracker

Client

Job Tracker

Task Tracker Task Tracker

Name Node
Data Data
Data Data
Node Node
Node Node

Only one Job Tracker per cluster

Receives job requests submitted by client

Schedules and monitors
Hadoop jobs on task trackers 20
 Hadoop Components: Name Node

Client
Job
Tracker
Task Task
Tracker Tracker
Name Node
Data Data
Data Data
Node Node
Node Node

One active Name Node per cluster

Manages the file system namespace and metadata

Single point of failure: Good place to spend money on hardware
Hadoop 21
 Name Node

• Master of HDFS
• Maintains and Manages data on Data Nodes
• High reliability Machine (can be even RAID)
• Expensive Hardware
• Stores NO data; Just holds Metadata!
• Secondary Name Node:
– Reads from RAM of Name Node and stores it to hard
disks periodically.
• Active & Passive Name Nodes from Gen2 Hadoop

Hadoop 22
 Hadoop Components: Task Tracker

Client
Job
Tracker
Task Task
Tracker Tracker
Name Node
Data Data
Data Data
Node Node
Node Node

There are typically a lot of task trackers

Responsible for executing operations

Reads blocks of data from data nodes
Hadoop 23
 Hadoop Components: Data Node

Client
Job
Tracker
Task Task
Tracker Tracker
Name Node
Data Data
Data Data
Node Node
Node Node

There are typically a lot of data nodes

Data nodes manage data blocks and serve them to clients

Data is replicated so failure is not a problem
Hadoop 24
 Data Nodes

• Slaves in HDFS
• Provides Data Storage
• Deployed on independent machines
• Responsible for serving Read/Write requests from
Client.
• The data processing is done on Data Nodes.

Hadoop 25
HDFS Architecture

Hadoop 26
 Hadoop Modes of Operation

Hadoop supports three modes of operation:

• Standalone
• Pseudo-Distributed
• Fully-Distributed

Hadoop 27
HDFS Operation

Hadoop 28
 HDFS Operation
• Client makes a Write request to Name Node
• Name Node responds with the information about
on available data nodes and where data to be
written.
• Client write the data to the addressed Data Node.
• Replicas for all blocks are automatically created
by Data Pipeline.
• If Write fails, Data Node will notify the Client
and get new location to write.
• If Write Completed Successfully,
Acknowledgement is given to Client
• Non-Posted Write by Hadoop
Hadoop 29
HDFS: File Write

Hadoop 30
HDFS: File Read

Hadoop 31
• HadoopHadoop:
Development Platform
Hadoop Stack
– User written code runs on system
– System appears to user as a single entity
– User does not need to worry about
distributed system
– Many system can run on top of Hadoop
• Allows further abstraction from system

Hadoop 32

Hive and Hadoop: Hive on
HBase are layers & top
HBase
of Hadoop

HBase & Hive are applications

Provide an interface to data on the HDFS

Other programs or applications may use Hive or
HBase as an intermediate layer
HBase
ZooKeeper

Hadoop 33
 Hadoop: Hive

• Hive
– Data warehousing application
– SQL like commands (HiveQL)
– Not a traditional relational database
– Scales horizontally with ease
– Supports massive amounts of data*

* Facebook has more than 15PB of information stored in it and imports 60TB each day (as of 2010)
Hadoop 34
 Hadoop: HBase

• HBase
– No SQL Like language
• Uses custom Java API for working with data
– Modeled after Google’s BigTable
– Random read/write operations allowed
– Multiple concurrent read/write operations allowed

Hadoop 35
 Hadoop MapReduce
Hadoop has it’s own implementation of MapReduce
Hadoop 1.0.4
API: http://hadoop.apache.org/docs/r1.0.4/api/
Tutorial: http://hadoop.apache.org/docs/r1.0.4/mapred_tutorial.html
Custom Serialization
Data Types
Writable/Comparable
Text vs String
LongWritable vs long
IntWritable vs int
DoubleWritable vs double

Hadoop 36
Structure of a Hadoop Mapper (WordCount)

Hadoop 37
Structure of a Hadoop Reducer (WordCount)

Hadoop 38
 Hadoop MapReduce

Working with the Hadoop

http://hadoop.apache.org/docs/r1.0.4/commands_manual.html

A quick overview of Hadoop commands

bin/start-all.sh

bin/stop-all.sh

bin/hadoop fs –put localSourcePath hdfsDestinationPath

bin/hadoop fs –get hdfsSourcePath localDestinationPath

bin/hadoop fs –rmr folderToDelete

bin/hadoop job –kill job_id

Running a Hadoop MR Program

bin/hadoop jar jarFileName.jar programToRun parm1 parm2…

SS Chung CIS 612 Lecture Notes 39

 Useful Application Sites
[1] http://wiki.apache.org/hadoop/EclipsePlugIn
[2] 10gen. Mongodb. http://www.mongodb.org/
[3] Apache. Cassandra. http://cassandra.apache.org/
[4] Apache. Hadoop. http://hadoop.apache.org/
[5] Apache. Hbase. http://hbase.apache.org/
[6] Apache, Hive. http://hive.apache.org/
[7] Apache, Pig. http://pig.apache.org/
[8] Zoo Keeper, http://zookeeper.apache.org/

Hadoop 40
How MapReduce Works in Hadoop
Lifecycle of a MapReduce Job

Map function

Reduce function

Run this program as a

MapReduce job
Lifecycle of a MapReduce Job

Map function

Reduce function

Run this program as a

MapReduce job
Lifecycle of a MapReduce Job
Time

Input Map Map Reduce Reduce

Splits Wave 1 Wave 2 Wave 1 Wave 2
Hadoop MR Job Interface:
Input Format
• The Hadoop MapReduce framework spawns
one map task for each InputSplit
• InputSplit: Input File is Split to Input Splits (Logical
splits (usually 1 block), not Physically split chunks)
Input Format::getInputSplits()
• The number of maps is usually driven by the total
number of blocks (InputSplits) of the input files.
1 block size = 128 MB,
10 TB file configured with 82000 maps
Hadoop MR Job Interface:
map()
• The framework then calls
map(WritableComparable, Writable, OutputCollector,
Reporter) for each key/value pair (line_num, line_string
) in the InputSplit for that task.
• Output pairs are collected with calls to
OutputCollector.collect(WritableComparable,Writable).
Hadoop MR Job Interface:
combiner()
• Optional combiner, via
JobConf.setCombinerClass(Class)
• to perform local aggregation of the intermediate
outputs of mapper
Hadoop MR Job Interface:
Partitioner()
• Partitioner controls the partitioning of the keys of the
intermediate map-outputs.
• The key (or a subset of the key) is used to derive the
partition, typically by a hash function.
• The total number of partitions is the same as the
number of reducers
• HashPartitioner is the default Partitioner of reduce
tasks for the job
Hadoop MR Job Interface:
reducer()
• Reducer has 3 primary phases:
1. Shuffle:
2. Sort
3. Reduce
Hadoop MR Job Interface:
reducer()
• Shuffle
Input to the Reducer is the sorted output of the mappers.
In this phase the framework fetches the relevant
partition of the output of all the mappers, via HTTP.
• Sort
The framework groups Reducer inputs by keys (since
different mappers may have output the same key) in this
stage.
• The shuffle and sort phases occur simultaneously;
while map-outputs are being fetched they are merged.
Hadoop MR Job Interface:
reducer()
• Reduce
The framework then calls
reduce(WritableComparable, Iterator, OutputCollector, Reporter)
method for each <key, (list of values)> pair in the
grouped inputs.
• The output of the reduce task is typically written to
the FileSystem via
OutputCollector.collect(WritableComparable, Writable).
 MR Job Parameters

• Map Parameters
io.sort.mb
• Shuffle/Reduce Parameters
io.sort.factor
mapred.inmem.merge.threshold
mapred.job.shuffle.merge.percent
 Components in a Hadoop MR Workflow

Next few slides are from: http://www.slideshare.net/hadoop/practical-problem-solving-with-apache-hadoop-pig

Job Submission
Initialization
Scheduling
Execution
Map Task
Sort Buffer
Reduce Tasks
 Quick Overview of Other Topics

• Dealing with failures

• Hadoop Distributed FileSystem (HDFS)
• Optimizing a MapReduce job
 Dealing with Failures and Slow Tasks
• What to do when a task fails?
– Try again (retries possible because of idempotence)
– Try again somewhere else
– Report failure
• What about slow tasks: stragglers
– Run another version of the same task in parallel. Take
results from the one that finishes first
– What are the pros and cons of this approach?

Fault tolerance is of
high priority in the
MapReduce framework
HDFS Architecture
 Lifecycle of a MapReduce Job
Time

Input Map Map Reduce Reduce

Splits Wave 1 Wave 2 Wave 1 Wave 2

How are the number of splits, number of map and reduce

tasks, memory allocation to tasks, etc., determined?
Job Configuration Parameters
• 190+ parameters in
Hadoop
• Set manually or defaults
are used
 Hadoop Job Configuration Parameters

Image source: http://www.jaso.co.kr/265

 Tuning Hadoop Job Conf. Parameters
• Do their settings impact performance?
• What are ways to set these parameters?
– Defaults -- are they good enough?
– Best practices -- the best setting can depend on data, job, and
cluster properties
– Automatic setting
 Experimental Setting
• Hadoop cluster on 1 master + 16 workers
• Each node:
– 2GHz AMD processor, 1.8GB RAM, 30GB local disk
– Relatively ill-provisioned!
– Xen VM running Debian Linux
– Max 4 concurrent maps & 2 reduces
• Maximum map wave size = 16x4 = 64
• Maximum reduce wave size = 16x2 = 32
• Not all users can run large Hadoop clusters:
– Can Hadoop be made competitive in the 10-25 node, multi GB
to TB data size range?
Parameters Varied in Experiments
 Hadoop 50GB TeraSort

• Varying number of reduce tasks, number of concurrent sorted

streams for merging, and fraction of map-side sort buffer
devoted to metadata storage
 Hadoop 50GB TeraSort

• Varying number of reduce tasks for different values

of the fraction of map-side sort buffer devoted to
metadata storage (with io.sort.factor = 500)
 Hadoop 50GB TeraSort

• Varying number of reduce tasks for different values of

io.sort.factor (io.sort.record.percent = 0.05, default)
Hadoop 75GB TeraSort

• 1D projection for
io.sort.factor=500
Automatic Optimization? (Not yet in Hadoop)

Shuffle
Map Map Map Reduce Reduce
Wave 1 Wave 2 Wave 3 Wave 1 Wave 2

What if
#reduces
increased
to 9?

Map Map Map Reduce Reduce Reduce

Wave 1 Wave 2 Wave 3 Wave 1 Wave 2 Wave 3