Assignment I

Q.1 How to import RDBMS table in Hadoop using Sqoop when the table doesn’t have a
primary key column?

Ans: Usually, we import an RDBMS table in Hadoop using Sqoop Importwhen it has a
primary key column. If it doesn’t have the primary key column, it will give you the below
error-
ERROR tool.ImportTool: Error during import: No primary key could be found for table
<table_name>. Please specify one with –split-by or perform a sequential import with ‘-m 1’

Here is the solution of what to do when you don’t have a primary key column in RDBMS,
and you want to import using Sqoop.

If your table doesn’t have the primary key column, you need to specify -m 1option for
importing the data, or you have to provide –split-by argument with some column name.
Here are the scripts which you can use to import an RDBMS table in Hadoop using Sqoop
when you don’t have a primary key column.

sqoop import \
–connect jdbc:mysql://localhost/dbname \
–username root \
–password root \
–table user \
–target-dir /user/root/user_data \
–columns “first_name, last_name, created_date”
-m 1

or

sqoop import \
–connect jdbc:mysql://localhost/ dbname\
–username root \
–password root \
–table user \
–target-dir /user/root/user_data \
–columns “first_name, last_name, created_date”
–split-by created_date

Q.2 Suppose I have installed Apache Hive on top of my Hadoop cluster using
default metastore configuration. Then, what will happen if we have multiple
clients trying to access Hive at the same time?

The default metastore configuration allows only one Hive session to be opened at a

time for accessing the metastore. Therefore, if multiple clients try to access the

metastore at the same time, they will get an error. One has to use a standalone

metastore, i.e. Local or remote metastore configuration in Apache Hive for allowing

access to multiple clients concurrently.

Following are the steps to configure MySQL database as the local metastore in Apache

Hive:

One should make the following changes in hive-site.xml:

javax.jdo.option.ConnectionURL property should be set to

jdbc:mysql://host/dbname?createDataba
seIfNotExist=true.
javax.jdo.option.ConnectionDriverName property should be set
to com.mysql.jdbc.Driver.
One should also set the username and password as:
javax.jdo.option.ConnectionUserName is set to desired username.
javax.jdo.option.ConnectionPassword is set to the desired
password.
The JDBC driver JAR file for MySQL must be on the Hive’s classpath, i.e. The jar
file should be copied into the Hive’s lib directory.

Now, after restarting the Hive shell, it will automatically connect to the MySQL
database which is running as a standalone metastore.
Q.3 Suppose, I create a table that contains details of all the transactions done by
the customers of year 2016: CREATE TABLE transaction_details (cust_id INT,
amount FLOAT, month STRING, country STRING) ROW FORMAT DELIMITED
FIELDS TERMINATED BY ‘,’ ;

Now, after inserting 50,000 tuples in this table, I want to know the total revenue

generated for each month. But, Hive is taking too much time in processing this

query. How will you solve this problem and list the steps that I will be taking in

order to do so?

We can solve this problem of query latency by partitioning the table according to each

month. So, for each month we will be scanning only the partitioned data instead of

whole data sets.

As we know, we can’t partition an existing non-partitioned table directly. So, we will be

taking following steps to solve the very problem:

1.Create a partitioned table, say partitioned_transaction:

CREATE TABLE partitioned_transaction (cust_id INT, amount FLOAT, country STRING)

PARTITIONED BY (month STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY

‘,’ ;

2. Enable dynamic partitioning in Hive:

SET hive.exec.dynamic.partition = true;

SET hive.exec.dynamic.partition.mode = nonstrict;

3. Transfer the data from the non – partitioned table into the newly created

partitioned table:

INSERT OVERWRITE TABLE partitioned_transaction PARTITION (month) SELECT cust_id,

amount, country, month FROM transaction_details;

Now, we can perform the query using each partition and therefore, decrease the

query time.

Q.4 Suppose, I have a lot of small CSV files present in /input directory in HDFS
and I want to create a single Hive table corresponding to these files. The data in
these files are in the format: {id, name, e-mail, country}. Now, as we know,
Hadoop performance degrades when we use lots of small files.

So, how will you solve this problem where we want to create a single Hive table

for lots of small files without degrading the performance of the system?

One can use the SequenceFile format which will group these small files together to

form a single sequence file. The steps that will be followed in doing so are as follows:

Create a temporary table:

CREATE TABLE temp_table (id INT, name STRING, e-mail STRING, country STRING)

ROW FORMAT FIELDS DELIMITED TERMINATED BY ‘,’ STORED AS TEXTFILE;

Load the data into temp_table:

LOAD DATA INPATH ‘/input’ INTO TABLE temp_table;

Create a table that will store data in SequenceFile format:

CREATE TABLE sample_seqfile (id INT, name STRING, e-mail STRING, country STRING)

ROW FORMAT FIELDS DELIMITED TERMINATED BY ‘,’ STORED AS SEQUENCEFILE;

Transfer the data from the temporary table into the sample_seqfile table:

INSERT OVERWRITE TABLE sample SELECT * FROM temp_table;

Hence, a single SequenceFile is generated which contains the data present in all of the

input files and therefore, the problem of having lots of small files is finally eliminated.

Assignment II

Q.5 Suppose, I have a CSV file – ‘sample.csv’ present in ‘/temp’ directory with the
following entries:
id first_name last_name email gender ip_address

1 Hugh Jackman hughjackman@cam.ac.uk Male 136.90.241.52

2 David Lawrence dlawrence1@gmail.com Male 101.177.15.130

3 Andy Hall andyhall2@yahoo.com Female 114.123.153.64

4 Samuel Jackson samjackson231@sun.com Male 89.60.227.31

5 Emily Rose rose.emily4@surveymonkey.com Female 119.92.21.19

Q.6 How will you consume this CSV file into the Hive warehouse using built

SerDe?

SerDe stands for serializer/deserializer. A SerDe allows us to convert the unstructured

bytes into a record that we can process using Hive. SerDes are implemented using

Java. Hive comes with several built-in SerDes and many other third-party SerDes are

also available.

Hive provides a specific SerDe for working with CSV files. We can use this SerDe for

the sample.csv by issuing following commands:

CREATE EXTERNAL TABLE sample

(id int, first_name string,

last_name string, email string,

gender string, ip_address string)

ROW FORMAT SERDE ‘org.apache.hadoop.hive.serde2.OpenCSVSerde’

STORED AS TEXTFILE LOCATION ‘/temp’;

Now, we can perform any query on the table ‘sample’:

SELECT first_name FROM sample WHERE gender = ‘male’;

Q.7 Define the difference between Hive and HBase?
Apache Hive is a data warehousing infrastructure built on top of Hadoop. It helps in

querying data stored in HDFS for analysis using Hive Query Language (HQL), which is a

SQL-like language, that gets translated into MapReduce jobs. Hive performs batch

processing on Hadoop.

Apache HBase is NoSQL key/value store which runs on top of HDFS. Unlike Hive,

HBase operations run in real-time on its database rather than MapReduce jobs.

HBase partitions the tables, and the tables are further splitted into column families.

Hive and HBase are two different Hadoop based technologies – Hive is an SQL-like

engine that runs MapReduce jobs, and HBase is a NoSQL key/value database of

Hadoop. We can use them together. Hive can be used for analytical queries while

HBase for real-time querying. Data can even be read and written from HBase to Hive

and vice-versa.

Q. 8 Explain Storage Mechanism in HBase

HBase is a column-oriented database and data is stored in tables. The

tables are sorted by RowId. As shown below, HBase has RowId, which
is the collection of several column families that are present in the
table.
The column families that are present in the schema are key-value
pairs. If we observe in detail each column family having multiple
numbers of columns. The column values stored into disk memory.
Each cell of the table has its own Metadata like timestamp and other
information.
Coming to HBase the following are the key terms representing table
schema
Table: Collection of rows present.
Row: Collection of column families.
Column Family: Collection of columns.
Column: Collection of key-value pairs.
Namespace: Logical grouping of tables.
Cell: A {row, column, version} tuple exactly specifies a cell
definition in HBase.

Column-oriented vs Row-oriented storages

Column and Row-oriented storages differ in their storage mechanism.
As we all know traditional relational models store data in terms of row-
based format like in terms of rows of data. Column-oriented storages
store data tables in terms of columns and column families.
The following Table gives some key differences between these two
storages
Column-oriented Database Row oriented Database
 When the situation comes to process and
 Online Transactional processsuch
analytics we use this approach. Such
as banking and finance domains
as Online Analytical Processing and it's
use this approach.
applications.
  The amount of data that can able to store in
 It is designed for a small number
this model is very huge like in terms of
of rows and columns.
petabytes

HBase Data Model

HBase Data Model consists of following elements,
Set of tables
Each table with column families and rows
Each table must have an element defined as Primary Key.
Row key acts as a Primary key in HBase.
Any access to HBase tables uses this Primary Key
Each column present in HBase denotes attribute corresponding
to object