Apache Hadoop Ecosystem

Components of Apache Hadoop

Apache Hadoop ecosystem is the set of services, which can be used
at a different level of big data processing and use by many
organizations to solve big data problems. HDFS and HBase are used
to store data, Spark and MapReduce are used to process data,
Flume and Sqoop are used to ingest data, Pig, Hive, and Impala are
used to analyze data, Hue and Cloudera Search help to explore
data. Oozie manages the workflow of Hadoop jobs and so on.

Let’s have a look at the Apache Hadoop Ecosystem.

1. HDFS (Hadoop Distributed File System)
The Apache Hadoop HDFS is the distributed file system of Hadoop
that is designed to store large files on cheap hardware. It is highly
fault-tolerant and provides high throughput to applications. HDFS is
best suited for those applications which are having very large data
sets.

The Hadoop HDFS file system provides Master and Slave

architecture. The Master node runs Name node daemons and Slave
nodes run Datanode daemons.

2. Map-Reduce
Map-Reduce is the data processing layer of Hadoop, It distributes
the task into small pieces and assigns those pieces to many
machines joined over a network, and assembles all the events to
form the last event dataset. The basic detail required by Map-
Reduce is a key-value pair. All the data, whether structured or not,
needs to be translated to the key-value pair before it is passed
through the Map-Reduce model. In the Map-Reduce Framework, the
processing unit is moved to the data rather than moving the data to
the processing unit.
3. YARN
YARN stands for “Yet Another Resource Negotiator” which is the
Resource Management level of the Hadoop Cluster. YARN is used to
implement resource management and job scheduling in the Hadoop
cluster. The primary idea of YARN is to split the job scheduling and
resource management into various processes and make the
operation.

YARN gives two daemons; the first one is called Resource Manager
and the seconds one is called Node Manager. Both components are
used to process data-computation in YARN. The Resource Manager
runs on the master node of the Hadoop cluster and negotiates
resources in all applications whereas the Node Manager is hosted on
all Slave nodes. The responsibility of the Node Manager is to monitor
the containers, resource usage such as (CPU, memory, disk, and
network) and provide detail to the Resource Manager.
4. Apache Hive
Apache Hive is a data-warehousing project of Hadoop. Hive is
intended to facilitate informal data summarization, ad-hoc querying,
and interpretation of large volumes of data. With the help of HiveQL,
a user can perform ad-hoc queries on the dataset store in HDFS and
use that data to do further analysis. Hive also supports custom user-
defined functions that can be used by users to perform custom
analysis.

Let us understand how Apache Hive processes the SQL query.

 Users will submit a query using the command line or Web UI to

the driver (Such as ODBC/JDBC).
 The driver program will take help from the query compiler that
parses the query to check the syntax/query plan.
 The compiler will send metadata requests to the metadata
database.
 In response, Metastore will provide metadata to the compiler.
 Now the task of the compiler is to verify the specification and
resend the plan to the driver.
 Now the driver will send an execution plan to the execution
engine.
  The program will be executed as a map-reduce job. The
execution engine will send the job to the name node job
tracker and it assigns this job a task tracker that is present in
the data node and here query will be executed.
 Post query execution, the execution engine will receive the
result from the data node.
 The execution engine will send the result value to the driver.
 The driver will send the result to the Hive interface (User).

5. Apache Pig
Apache Pig was developed by Yahoo to analyze large data stored in
Hadoop HDFS. Pig provides a platform to analyze massive data sets
that consists of a high-level language for communicating data
analysis applications, linked with infrastructure for assessing these
programs.

Apache Pig has the following key properties.

Optimization Opportunities

Apache Pig provides the optimization of the query that helps users
to concentrate on meaning rather than efficiency.
Extensibility

Apache Pig provides functionality to create User-defined functions to

produce special-purpose processing.

6. Apache Mahout
Apache Mahout is a framework to create machine-learning
applications. It provides a rich set of components from which you
can construct a customized recommender system from a selection
of algorithms. Mahout is developed to provide enforcement,
scalability, and compliance.

The following are the important packages that define the Mahout
interfaces to these key abstractions.

 DataModel
 UserSimilarity
 ItemSimilarity
 UserNeighborhood
 Recommender
7. Apache HBase
Apache HBase is a distributed, open-source, versioned, and non-
relational database that is created after Google's Bigtable. It is an
import component of the Hadoop ecosystem that leverages the fault
tolerance feature of HDFS and it provides real-time read and writes
access to data. Hbase can be called a data storage system despite a
database because it doesn’t provide RDBMS features like triggers,
query language, and secondary indexes.

Apache HBase has the following features.

 It provides continuing and modular scalability.

 It surely provides regular reads and writes.
 Intuitive and configurable sharding of tables.
 Automated failover support between RegionServers.
  It provides Central base classes for supporting Hadoop
MapReduce jobs with Apache HBase tables.
 It is simple to use Java API for client access.
 Query predicate pushes down via server-side Filter.
 It provides a Thrift gateway and a REST-ful Web service that
supports XML, Protobuf, and binary data encoding choices.

8. Apache Zookeeper
Apache Zookeeper acts as a coordinator between different services
of Hadoop and is used for maintaining configuration information,
naming, providing distributed synchronization, and providing group
services. Zookeeper is used to fix bugs and race conditions for those
applications, which are newly deployed in a distributed environment.
9. Apache Sqoop
Apache Sqoop is a data transfer tool, which is used to transfer data
between Hadoop and relational databases. It is used to import data
from a relational database management system like (MySQL or
Oracle) or a mainframe into the Hadoop (HDFS), transform the data
in Hadoop MapReduce. It is also used to export the data back to an
RDBMS. Sqoop uses map-reduce to import and export data due to
which it gets parallel processing and fault-tolerance property.
10. Apache Flume
Apache Flume is a log transfer tool similar to Sqoop but it works on
unstructured data (logs) whereas Sqoop is used for structure and
unstructured data. Flume is a reliable, distributed, and available
system for efficiently collecting, aggregating, and moving large
amounts of log data from many different sources to an HDFS. It is
not restricted to log data aggregation only but it can also use to
transport massive quantities of event data.

Apache Flume has the following three components.

 Source
 Channel
 Sink

11. Apache Oozie

Apache Oozie is a workflow scheduling framework that is used to
schedule Hadoop Map/Reduce and Pig jobs. An Apache Oozie
workflow is the collection of actions such as Hadoop Map/Reduce
jobs, Pig jobs, that is arranged in a control dependency DAG (Direct
Acyclic Graph). The "control dependency" from one action to
another suggests that the other action will not start unless the first
action has been completed.

Apache Oozie workflow has the following two nodes namely Control
Flow Nodes and Action Node.

1. Control Flow Nodes

These nodes are used to provide a mechanism to control the

workflow execution path.

2. Action Node

Action node provides a mechanism by which a workflow triggers the

execution of a computation/processing task such as "Hadoop map-
reduce, Hadoop file system, Pig, SSH, HTTP jobs".

12. Apache Ambari

Apache Ambari is used for provisioning, managing, and monitoring
Apache Hadoop clusters.

It offers the below task to the system admin.

1. Provisioning of a Hadoop Cluster

It provides a medium to install the Hadoop services over any

number of nodes. It also handles the configuration of Hadoop
services for a cluster.

2. Management of a Hadoop Cluster

It provides a central control to manage Hadoop services like

starting, stopping, and reconfiguring across the entire cluster.

3. Monitoring of a Hadoop Cluster

It provides a dashboard for monitoring of the Hadoop cluster (like a

node down, remaining disk space is low, etc).

13. Apache Spark

Apache Spark is a general-purpose and fast cluster computing
system. It is a very powerful tool for Big data. Spark provides a rich
set of APIs in multiple languages such as Python, Scala, Java, R, and
so on. Spark supports high-level tools which are Spark SQL, GraphX,
MLlib, Spark Streaming, R. These tools are used to perform a
different kind of operation which we will see in the Apache Spark
section.