Pig Latin is the language used to analyze data in Hadoop using Apache Pig.

In this chapter,
we are going to discuss the basics of Pig Latin such as Pig Latin statements, data types,
general and relational operators, and Pig Latin UDF‟s.

Pig Latin – Data Model

As discussed in the previous chapters, the data model of Pig is fully nested. A Relation is
the outermost structure of the Pig Latin data model. And it is a bag where −

 A bag is a collection of tuples.

 A tuple is an ordered set of fields.
 A field is a piece of data.

Pig Latin – Statements

While processing data using Pig Latin, statements are the basic constructs.
 These statements work with relations. They include expressions and schemas.
 Every statement ends with a semicolon (;).
 We will perform various operations using operators provided by Pig Latin, through
statements.
 Except LOAD and STORE, while performing all other operations, Pig Latin
statements take a relation as input and produce another relation as output.
 As soon as you enter a Load statement in the Grunt shell, its semantic checking will
be carried out. To see the contents of the schema, you need to use
the Dump operator. Only after performing the dump operation, the Map Reduce job
for loading the data into the file system will be carried out.

Example

Given below is a Pig Latin statement, which loads data to Apache Pig.
grunt> Student_data = LOAD 'student_data.txt' USING PigStorage(',')as
( id:int, firstname:chararray, lastname:chararray, phone:chararray, city:chararray );

Pig Latin – Data types

Given below table describes the Pig Latin data types.

S.N. Data Type Description & Example

1 int Represents a signed 32-bit integer.

Example : 8
2 long Represents a signed 64-bit integer.
Example : 5L

3 float Represents a signed 32-bit floating point.

Example : 5.5F

4 double Represents a 64-bit floating point.

Example : 10.5

5 chararray Represents a character array (string) in Unicode UTF-8 format.

Example : „Big Data‟

6 Bytearray Represents a Byte array (blob).

7 Boolean Represents a Boolean value.

Example : true/ false.

8 Datetime Represents a date-time.

Example : 1970-01-01T00:00:00.000+00:00

9 Biginteger Represents a Java BigInteger.

Example : 60708090709

10 Bigdecimal Represents a Java BigDecimal

Example : 185.98376256272893883

Complex Types

11 Tuple A tuple is an ordered set of fields.

Example : (raja, 30)

12 Bag
A bag is a collection of tuples.
Example : {(raju,30),(Mohhammad,45)}
13 Map A Map is a set of key-value pairs.
Example : [ „name‟#‟Raju‟, „age‟#30]

Null Values
Values for all the above data types can be NULL. Apache Pig treats null values in a similar
way as SQL does.
A null can be an unknown value or a non-existent value. It is used as a placeholder for
optional values. These nulls can occur naturally or can be the result of an operation.

Pig Latin – Arithmetic Operators

The following table describes the arithmetic operators of Pig Latin. Suppose a = 10 and b =
20.

Operator Description Example

+ a + b will give 30
Addition − Adds values on either side of the operator

− a − b will give −10

Subtraction − Subtracts right hand operand from left hand
operand

* a * b will give 200

Multiplication − Multiplies values on either side of the
operator

/ b / a will give 2
Division − Divides left hand operand by right hand operand

% b % a will give 0
Modulus − Divides left hand operand by right hand operand
and returns remainder

Bincond − Evaluates the Boolean operators. It has three b = (a == 1)? 20: 30;
operands as shown below.
if a = 1 the value of
?: variable x = (expression) ? value1 if true : value2 if false. b is 20.
if a!=1 the value of b
is 30.

CASE Case − The case operator is equivalent to nested bincond CASE f2 % 2

WHEN operator. WHEN 0 THEN
'even'
THEN
WHEN 1 THEN
ELSE
'odd'
END
END

Pig Latin – Comparison Operators

The following table describes the comparison operators of Pig Latin.

Operator Description Example

== (a = b) is not
Equal − Checks if the values of two operands are equal or not; if
true
yes, then the condition becomes true.

!= (a != b) is true.
Not Equal − Checks if the values of two operands are equal or not.
If the values are not equal, then condition becomes true.

> (a > b) is not

Greater than − Checks if the value of the left operand is greater
true.
than the value of the right operand. If yes, then the condition
becomes true.

< (a < b) is true.

Less than − Checks if the value of the left operand is less than the
value of the right operand. If yes, then the condition becomes true.

>= (a >= b) is not

Greater than or equal to − Checks if the value of the left operand
true.
is greater than or equal to the value of the right operand. If yes, then
the condition becomes true.

<= (a <= b) is true.

Less than or equal to − Checks if the value of the left operand is
less than or equal to the value of the right operand. If yes, then the
condition becomes true.

matches f1 matches
Pattern matching − Checks whether the string in the left-hand side
'.*tutorial.*'
matches with the constant in the right-hand side.

Pig Latin – Type Construction Operators

The following table describes the Type construction operators of Pig Latin.
Operator Description Example

() (Raju, 30)
Tuple constructor operator − This operator is used to
construct a tuple.

{} {(Raju, 30), (Mohammad,

Bag constructor operator − This operator is used to
45)}
construct a bag.

[] [name#Raja, age#30]
Map constructor operator − This operator is used to
construct a tuple.

Pig Latin – Relational Operations

The following table describes the relational operators of Pig Latin.

Operator Description

Loading and Storing

LOAD To Load the data from the file system (local/HDFS) into a relation.

STORE To save a relation to the file system (local/HDFS).

Filtering

FILTER To remove unwanted rows from a relation.

DISTINCT To remove duplicate rows from a relation.

FOREACH, To generate data transformations based on columns of data.

GENERATE

STREAM To transform a relation using an external program.

 Grouping and Joining

JOIN To join two or more relations.

COGROUP To group the data in two or more relations.

GROUP To group the data in a single relation.

CROSS To create the cross product of two or more relations.

Sorting

ORDER To arrange a relation in a sorted order based on one or more fields

(ascending or descending).

LIMIT To get a limited number of tuples from a relation.

Combining and Splitting

UNION To combine two or more relations into a single relation.

SPLIT To split a single relation into two or more relations.

Diagnostic Operators

DUMP To print the contents of a relation on the console.

DESCRIBE To describe the schema of a relation.

EXPLAIN To view the logical, physical, or MapReduce execution plans to

compute a relation.

ILLUSTRATE To view the step-by-step execution of a series of statements.

 Apache Pig - Reading Data
In general, Apache Pig works on top of Hadoop. It is an analytical tool that analyzes large datasets
that exist in the Hadoop File System. To analyze data using Apache Pig, we have to initially load the
data into Apache Pig. This chapter explains how to load data to Apache Pig from HDFS.

Preparing HDFS
In MapReduce mode, Pig reads (loads) data from HDFS and stores the results back in HDFS.
Therefore, let us start HDFS and create the following sample data in HDFS.

Student ID First Name Last Name Phone City

001 Rajiv Reddy 9848022337 Hyderabad

002 siddarth Battacharya 9848022338 Kolkata

003 Rajesh Khanna 9848022339 Delhi

004 Preethi Agarwal 9848022330 Pune

005 Trupthi Mohanthy 9848022336 Bhuwaneshwar

006 Archana Mishra 9848022335 Chennai

The above dataset contains personal details like id, first name, last name, phone number and city, of
six students.

Step 1: Verifying Hadoop

First of all, verify the installation using Hadoop version command, as shown below.
$ hadoop version

If your system contains Hadoop, and if you have set the PATH variable, then you will get the
following output −
Hadoop 2.6.0
Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r
e3496499ecb8d220fba99dc5ed4c99c8f9e33bb1
Compiled by jenkins on 2014-11-13T21:10Z
Compiled with protoc 2.5.0
From source with checksum 18e43357c8f927c0695f1e9522859d6a
This command was run using /home/Hadoop/hadoop/share/hadoop/common/hadoop
common-2.6.0.jar

Step 2: Starting HDFS

Browse through the sbin directory of Hadoop and start yarn and Hadoop dfs (distributed file system)
as shown below.
cd /$Hadoop_Home/sbin/
$ start-dfs.sh
localhost: starting namenode, logging to /home/Hadoop/hadoop/logs/hadoopHadoop-namenode-
localhost.localdomain.out
localhost: starting datanode, logging to /home/Hadoop/hadoop/logs/hadoopHadoop-datanode-
localhost.localdomain.out
Starting secondary namenodes [0.0.0.0]
starting secondarynamenode, logging to /home/Hadoop/hadoop/logs/hadoop-Hadoopsecondarynamenode-
localhost.localdomain.out

$ start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /home/Hadoop/hadoop/logs/yarn-Hadoopresourcemanager-
localhost.localdomain.out
localhost: starting nodemanager, logging to /home/Hadoop/hadoop/logs/yarnHadoop-nodemanager-
localhost.localdomain.out

Step 3: Create a Directory in HDFS

In Hadoop DFS, you can create directories using the command mkdir. Create a new directory in
HDFS with the name Pig_Data in the required path as shown below.
$cd /$Hadoop_Home/bin/
$ hdfs dfs -mkdir hdfs://localhost:9000/Pig_Data

Step 4: Placing the data in HDFS

The input file of Pig contains each tuple/record in individual lines. And the entities of the record are
separated by a delimiter (In our example we used “,”).
In the local file system, create an input file student_data.txt containing data as shown below.
001,Rajiv,Reddy,9848022337,Hyderabad
002,siddarth,Battacharya,9848022338,Kolkata
003,Rajesh,Khanna,9848022339,Delhi
004,Preethi,Agarwal,9848022330,Pune
005,Trupthi,Mohanthy,9848022336,Bhuwaneshwar
006,Archana,Mishra,9848022335,Chennai.
Now, move the file from the local file system to HDFS using put command as shown below. (You
can use copyFromLocal command as well.)
$ cd $HADOOP_HOME/bin
$ hdfs dfs -put /home/Hadoop/Pig/Pig_Data/student_data.txt dfs://localhost:9000/pig_data/

Verifying the file

You can use the cat command to verify whether the file has been moved into the HDFS, as shown
below.
$ cd $HADOOP_HOME/bin
$ hdfs dfs -cat hdfs://localhost:9000/pig_data/student_data.txt

Output
You can see the content of the file as shown below.
15/10/01 12:16:55 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable

001,Rajiv,Reddy,9848022337,Hyderabad
002,siddarth,Battacharya,9848022338,Kolkata
003,Rajesh,Khanna,9848022339,Delhi
004,Preethi,Agarwal,9848022330,Pune
005,Trupthi,Mohanthy,9848022336,Bhuwaneshwar
006,Archana,Mishra,9848022335,Chennai

The Load Operator

You can load data into Apache Pig from the file system (HDFS/ Local) using LOAD operator of Pig
Latin.

Syntax

The load statement consists of two parts divided by the “=” operator. On the left-hand side, we need
to mention the name of the relation where we want to store the data, and on the right-hand side, we
have to define how we store the data. Given below is the syntax of the Load operator.
Relation_name = LOAD 'Input file path' USING function as schema;
Where,
 relation_name − We have to mention the relation in which we want to store the data.
 Input file path − We have to mention the HDFS directory where the file is stored. (In
MapReduce mode)
 function − We have to choose a function from the set of load functions provided by Apache
Pig (BinStorage, JsonLoader, PigStorage, TextLoader).
 Schema − We have to define the schema of the data. We can define the required schema as
follows −
(column1 : data type, column2 : data type, column3 : data type);
Note − We load the data without specifying the schema. In that case, the columns will be addressed
as $01, $02, etc… (check).

Example

As an example, let us load the data in student_data.txt in Pig under the schema
named Student using the LOAD command.

Start the Pig Grunt Shell

First of all, open the Linux terminal. Start the Pig Grunt shell in MapReduce mode as shown below.
$ Pig –x mapreduce

It will start the Pig Grunt shell as shown below.

15/10/01 12:33:37 INFO pig.ExecTypeProvider: Trying ExecType : LOCAL
15/10/01 12:33:37 INFO pig.ExecTypeProvider: Trying ExecType : MAPREDUCE
15/10/01 12:33:37 INFO pig.ExecTypeProvider: Picked MAPREDUCE as the ExecType
2015-10-01 12:33:38,080 [main] INFO org.apache.pig.Main - Apache Pig version 0.15.0 (r1682971) compiled
Jun 01 2015, 11:44:35
2015-10-01 12:33:38,080 [main] INFO org.apache.pig.Main - Logging error messages to:
/home/Hadoop/pig_1443683018078.log
2015-10-01 12:33:38,242 [main] INFO org.apache.pig.impl.util.Utils - Default bootup file
/home/Hadoop/.pigbootup not found

2015-10-01 12:33:39,630 [main]

INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system
at: hdfs://localhost:9000

grunt>

Execute the Load Statement

Now load the data from the file student_data.txt into Pig by executing the following Pig Latin
statement in the Grunt shell.
grunt> student = LOAD 'hdfs://localhost:9000/pig_data/student_data.txt'
USING PigStorage(',')
as ( id:int, firstname:chararray, lastname:chararray, phone:chararray,
city:chararray );

Following is the description of the above statement.

Relation We have stored the data in the schema student.

name

Input file We are reading data from the file student_data.txt, which is in the /pig_data/
path directory of HDFS.

Storage We have used the PigStorage() function. It loads and stores data as structured text
function files. It takes a delimiter using which each entity of a tuple is separated, as a
parameter. By default, it takes „\t‟ as a parameter.

schema
We have stored the data using the following schema.

column id firstname lastname phone city

datatype int char array char array char array char array

Note − The load statement will simply load the data into the specified relation in Pig. To verify the
execution of the Load statement, you have to use the Diagnostic Operators which are discussed in
the next chapters.
 Apache Pig - Storing Data
In the previous chapter, we learnt how to load data into Apache Pig. You can store the loaded data in
the file system using the store operator. This chapter explains how to store data in Apache Pig using
the Store operator.

Syntax
Given below is the syntax of the Store statement.
STORE Relation_name INTO ' required_directory_path ' [USING function];

Example
Assume we have a file student_data.txt in HDFS with the following content.
001,Rajiv,Reddy,9848022337,Hyderabad
002,siddarth,Battacharya,9848022338,Kolkata
003,Rajesh,Khanna,9848022339,Delhi
004,Preethi,Agarwal,9848022330,Pune
005,Trupthi,Mohanthy,9848022336,Bhuwaneshwar
006,Archana,Mishra,9848022335,Chennai.
And we have read it into a relation student using the LOAD operator as shown below.
grunt> student = LOAD 'hdfs://localhost:9000/pig_data/student_data.txt'
USING PigStorage(',')
as ( id:int, firstname:chararray, lastname:chararray, phone:chararray,
city:chararray );

Now, let us store the relation in the HDFS directory “/pig_Output/” as shown below.
grunt> STORE student INTO ' hdfs://localhost:9000/pig_Output/ ' USING PigStorage (',');

Output

After executing the store statement, you will get the following output. A directory is created with the
specified name and the data will be stored in it.
2015-10-05 13:05:05,429 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.
MapReduceLau ncher - 100% complete
2015-10-05 13:05:05,429 [main] INFO org.apache.pig.tools.pigstats.mapreduce.SimplePigStats -
Script Statistics:

HadoopVersion PigVersion UserId StartedAt FinishedAt Features

2.6.0 0.15.0 Hadoop 2015-10-0 13:03:03 2015-10-05 13:05:05 UNKNOWN
Success!
Job Stats (time in seconds):
JobId Maps Reduces MaxMapTime MinMapTime AvgMapTime MedianMapTime
job_14459_06 1 0 n/a n/a n/a n/a
MaxReduceTime MinReduceTime AvgReduceTime MedianReducetime Alias Feature
0 0 0 0 student MAP_ONLY
OutPut folder
hdfs://localhost:9000/pig_Output/

Input(s): Successfully read 0 records from: "hdfs://localhost:9000/pig_data/student_data.txt"

Output(s): Successfully stored 0 records in: "hdfs://localhost:9000/pig_Output"
Counters:
Total records written : 0
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0

Job DAG: job_1443519499159_0006

2015-10-05 13:06:06,192 [main] INFO org.apache.pig.backend.hadoop.executionengine

.mapReduceLayer.MapReduceLau ncher - Success!

Verification
You can verify the stored data as shown below.

Step 1

First of all, list out the files in the directory named pig_output using the ls command as shown
below.
hdfs dfs -ls 'hdfs://localhost:9000/pig_Output/'
Found 2 items
rw-r--r- 1 Hadoop supergroup 0 2015-10-05 13:03 hdfs://localhost:9000/pig_Output/_SUCCESS
rw-r--r- 1 Hadoop supergroup 224 2015-10-05 13:03 hdfs://localhost:9000/pig_Output/part-m-00000
You can observe that two files were created after executing the store statement.

Step 2

Using cat command, list the contents of the file named part-m-00000 as shown below.
$ hdfs dfs -cat 'hdfs://localhost:9000/pig_Output/part-m-00000'
1,Rajiv,Reddy,9848022337,Hyderabad
2,siddarth,Battacharya,9848022338,Kolkata
3,Rajesh,Khanna,9848022339,Delhi
4,Preethi,Agarwal,9848022330,Pune
5,Trupthi,Mohanthy,9848022336,Bhuwaneshwar
6,Archana,Mishra,9848022335,Chennai

Apache Pig - Diagnostic Operators

The load statement will simply load the data into the specified relation in Apache Pig. To verify the
execution of the Load statement, you have to use the Diagnostic Operators. Pig Latin provides four
different types of diagnostic operators −

 Dump operator
 Describe operator
 Explanation operator
 Illustration operator
In this chapter, we will discuss the Dump operators of Pig Latin.

Dump Operator
The Dump operator is used to run the Pig Latin statements and display the results on the screen. It is
generally used for debugging Purpose.

Syntax

Given below is the syntax of the Dump operator.

grunt> Dump Relation_Name

Example

Assume we have a file student_data.txt in HDFS with the following content.

001,Rajiv,Reddy,9848022337,Hyderabad
002,siddarth,Battacharya,9848022338,Kolkata
003,Rajesh,Khanna,9848022339,Delhi
004,Preethi,Agarwal,9848022330,Pune
005,Trupthi,Mohanthy,9848022336,Bhuwaneshwar
006,Archana,Mishra,9848022335,Chennai.
And we have read it into a relation student using the LOAD operator as shown below.
grunt> student = LOAD 'hdfs://localhost:9000/pig_data/student_data.txt'
USING PigStorage(',')
as ( id:int, firstname:chararray, lastname:chararray, phone:chararray,
city:chararray );

Now, let us print the contents of the relation using the Dump operator as shown below.
grunt> Dump student
Once you execute the above Pig Latin statement, it will start a MapReduce job to read data from
HDFS. It will produce the following output.
2015-10-01 15:05:27,642 [main]
INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher -
100% complete
2015-10-01 15:05:27,652 [main]
INFO org.apache.pig.tools.pigstats.mapreduce.SimplePigStats - Script Statistics:
HadoopVersion PigVersion UserId StartedAt FinishedAt Features
2.6.0 0.15.0 Hadoop 2015-10-01 15:03:11 2015-10-01 05:27 UNKNOWN

Success!
Job Stats (time in seconds):

JobId job_14459_0004
Maps 1
Reduces 0
MaxMapTime n/a
MinMapTime n/a
AvgMapTime n/a
MedianMapTime n/a
MaxReduceTime 0
MinReduceTime 0
AvgReduceTime 0
MedianReducetime 0
Alias student
Feature MAP_ONLY
Outputs hdfs://localhost:9000/tmp/temp580182027/tmp757878456,

Input(s): Successfully read 0 records from: "hdfs://localhost:9000/pig_data/

student_data.txt"

Output(s): Successfully stored 0 records in: "hdfs://localhost:9000/tmp/temp580182027/

tmp757878456"

Counters: Total records written : 0 Total bytes written : 0 Spillable Memory Manager
spill count : 0Total bags proactively spilled: 0 Total records proactively spilled: 0

Job DAG: job_1443519499159_0004

2015-10-01 15:06:28,403 [main]

INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLau ncher - Success!
2015-10-01 15:06:28,441 [main] INFO org.apache.pig.data.SchemaTupleBackend -
Key [pig.schematuple] was not set... will not generate code.
2015-10-01 15:06:28,485 [main]
INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths
to process : 1
2015-10-01 15:06:28,485 [main]
INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths
to process : 1

(1,Rajiv,Reddy,9848022337,Hyderabad)
(2,siddarth,Battacharya,9848022338,Kolkata)
(3,Rajesh,Khanna,9848022339,Delhi)
(4,Preethi,Agarwal,9848022330,Pune)
(5,Trupthi,Mohanthy,9848022336,Bhuwaneshwar)
(6,Archana,Mishra,9848022335,Chennai)

Apache Pig - Describe Operator

The describe operator is used to view the schema of a relation.

Syntax
The syntax of the describe operator is as follows −
grunt> Describe Relation_name

Example
Assume we have a file student_data.txt in HDFS with the following content.
001,Rajiv,Reddy,9848022337,Hyderabad
002,siddarth,Battacharya,9848022338,Kolkata
003,Rajesh,Khanna,9848022339,Delhi
004,Preethi,Agarwal,9848022330,Pune
005,Trupthi,Mohanthy,9848022336,Bhuwaneshwar
006,Archana,Mishra,9848022335,Chennai.
And we have read it into a relation student using the LOAD operator as shown below.
grunt> student = LOAD 'hdfs://localhost:9000/pig_data/student_data.txt' USING PigStorage(',')
as ( id:int, firstname:chararray, lastname:chararray, phone:chararray, city:chararray );
Now, let us describe the relation named student and verify the schema as shown below.
grunt> describe student;

Output
Once you execute the above Pig Latin statement, it will produce the following output.
grunt> student: { id: int,firstname: chararray,lastname: chararray,phone: chararray,ci}

Apache Pig - Explain Operator

The explain operator is used to display the logical, physical, and MapReduce execution plans of a
relation.

Syntax
Given below is the syntax of the explain operator.
grunt> explain Relation_name;

Example
Assume we have a file student_data.txt in HDFS with the following content.
001,Rajiv,Reddy,9848022337,Hyderabad
002,siddarth,Battacharya,9848022338,Kolkata
003,Rajesh,Khanna,9848022339,Delhi
004,Preethi,Agarwal,9848022330,Pune
005,Trupthi,Mohanthy,9848022336,Bhuwaneshwar
006,Archana,Mishra,9848022335,Chennai.
And we have read it into a relation student using the LOAD operator as shown below.
grunt> student = LOAD 'hdfs://localhost:9000/pig_data/student_data.txt' USING PigStorage(',')
as ( id:int, firstname:chararray, lastname:chararray, phone:chararray, city:chararray );

Now, let us explain the relation named student using the explain operator as shown below.
grunt> explain student;

Output
It will produce the following output.
$ explain student;

2015-10-05 11:32:43,660 [main]

2015-10-05 11:32:43,660 [main] INFO org.apache.pig.newplan.logical.optimizer
.LogicalPlanOptimizer -
{RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, ConstantCalculator,
GroupByConstParallelSetter, LimitOptimizer, LoadTypeCastInserter, MergeFilter,
MergeForEach, PartitionFilterOptimizer, PredicatePushdownOptimizer,
PushDownForEachFlatten, PushUpFilter, SplitFilter, StreamTypeCastInserter]}
#-----------------------------------------------
# New Logical Plan:
#-----------------------------------------------
student: (Name: LOStore Schema:
id#31:int,firstname#32:chararray,lastname#33:chararray,phone#34:chararray,city#
35:chararray)
|
|---student: (Name: LOForEach Schema:
id#31:int,firstname#32:chararray,lastname#33:chararray,phone#34:chararray,city#
35:chararray)
| |
| (Name: LOGenerate[false,false,false,false,false] Schema:
id#31:int,firstname#32:chararray,lastname#33:chararray,phone#34:chararray,city#
35:chararray)ColumnPrune:InputUids=[34, 35, 32, 33,
31]ColumnPrune:OutputUids=[34, 35, 32, 33, 31]
| | |
| | (Name: Cast Type: int Uid: 31)
| | | | | |---id:(Name: Project Type: bytearray Uid: 31 Input: 0 Column: (*))
| | |
| | (Name: Cast Type: chararray Uid: 32)
| | |
| | |---firstname:(Name: Project Type: bytearray Uid: 32 Input: 1
Column: (*))
| | |
| | (Name: Cast Type: chararray Uid: 33)
| | |
| | |---lastname:(Name: Project Type: bytearray Uid: 33 Input: 2
Column: (*))
| | |
| | (Name: Cast Type: chararray Uid: 34)
| | |
| | |---phone:(Name: Project Type: bytearray Uid: 34 Input: 3 Column:
(*))
| | |
| | (Name: Cast Type: chararray Uid: 35)
| | |
| | |---city:(Name: Project Type: bytearray Uid: 35 Input: 4 Column:
(*))
| |
| |---(Name: LOInnerLoad[0] Schema: id#31:bytearray)
| |
| |---(Name: LOInnerLoad[1] Schema: firstname#32:bytearray)
| |
| |---(Name: LOInnerLoad[2] Schema: lastname#33:bytearray)
| |
| |---(Name: LOInnerLoad[3] Schema: phone#34:bytearray)
| |
| |---(Name: LOInnerLoad[4] Schema: city#35:bytearray)
|
|---student: (Name: LOLoad Schema:
id#31:bytearray,firstname#32:bytearray,lastname#33:bytearray,phone#34:bytearray
,city#35:bytearray)RequiredFields:null
#-----------------------------------------------
# Physical Plan: #-----------------------------------------------
student: Store(fakefile:org.apache.pig.builtin.PigStorage) - scope-36
|
|---student: New For Each(false,false,false,false,false)[bag] - scope-35
| |
| Cast[int] - scope-21
| |
| |---Project[bytearray][0] - scope-20
| |
| Cast[chararray] - scope-24
 | |
| |---Project[bytearray][1] - scope-23
| |
| Cast[chararray] - scope-27
| |
| |---Project[bytearray][2] - scope-26
| |
| Cast[chararray] - scope-30
| |
| |---Project[bytearray][3] - scope-29
| |
| Cast[chararray] - scope-33
| |
| |---Project[bytearray][4] - scope-32
|
|---student: Load(hdfs://localhost:9000/pig_data/student_data.txt:PigStorage(',')) - scope19
2015-10-05 11:32:43,682 [main]
INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler -
File concatenation threshold: 100 optimistic? false
2015-10-05 11:32:43,684 [main]
INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOp timizer -
MR plan size before optimization: 1 2015-10-05 11:32:43,685 [main]
INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.
MultiQueryOp timizer - MR plan size after optimization: 1
#--------------------------------------------------
# Map Reduce Plan
#--------------------------------------------------
MapReduce node scope-37
Map Plan
student: Store(fakefile:org.apache.pig.builtin.PigStorage) - scope-36
|
|---student: New For Each(false,false,false,false,false)[bag] - scope-35
| |
| Cast[int] - scope-21
| |
| |---Project[bytearray][0] - scope-20
| |
| Cast[chararray] - scope-24
| |
| |---Project[bytearray][1] - scope-23
| |
| Cast[chararray] - scope-27
| |
| |---Project[bytearray][2] - scope-26
| |
| Cast[chararray] - scope-30
| |
| |---Project[bytearray][3] - scope-29
| |
| Cast[chararray] - scope-33
| |
| |---Project[bytearray][4] - scope-32
|
|---student:
Load(hdfs://localhost:9000/pig_data/student_data.txt:PigStorage(',')) - scope
19-------- Global sort: false
----------------
 Apache Pig - Illustrate Operator
The illustrate operator gives you the step-by-step execution of a sequence of statements.

Syntax
Given below is the syntax of the illustrate operator.
grunt> illustrate Relation_name;

Example
Assume we have a file student_data.txt in HDFS with the following content.
001,Rajiv,Reddy,9848022337,Hyderabad
002,siddarth,Battacharya,9848022338,Kolkata
003,Rajesh,Khanna,9848022339,Delhi
004,Preethi,Agarwal,9848022330,Pune
005,Trupthi,Mohanthy,9848022336,Bhuwaneshwar
006,Archana,Mishra,9848022335,Chennai.
And we have read it into a relation student using the LOAD operator as shown below.
grunt> student = LOAD 'hdfs://localhost:9000/pig_data/student_data.txt' USING PigStorage(',')
as ( id:int, firstname:chararray, lastname:chararray, phone:chararray, city:chararray );

Now, let us illustrate the relation named student as shown below.

grunt> illustrate student;

Output
On executing the above statement, you will get the following output.
grunt> illustrate student;

INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$M ap - Aliases

being processed per job phase (AliasName[line,offset]): M: student[1,10] C: R:
----------------------------------------------------------------------------------------- ----
|student | id:int | firstname:chararray | lastname:chararray | phone:chararray | city:chararray |
---------------------------------------------------------------------------------------------
| | 002 | siddarth | Battacharya | 9848022338 | Kolkata |
---------------------------------------------------------------------------------
 Apache Pig - Group Operator

The GROUP operator is used to group the data in one or more relations. It collects the data having
the same key.

Syntax
Given below is the syntax of the group operator.
grunt> Group_data = GROUP Relation_name BY Key;

Example
Assume that we have a file named student_details.txt in the HDFS directory /pig_data/ as shown
below.
student_details.txt
001,Rajiv,Reddy,21,9848022337,Hyderabad
002,siddarth,Battacharya,22,9848022338,Kolkata
003,Rajesh,Khanna,22,9848022339,Delhi
004,Preethi,Agarwal,21,9848022330,Pune
005,Trupthi,Mohanthy,23,9848022336,Bhuwaneshwar
006,Archana,Mishra,23,9848022335,Chennai
007,Komal,Nayak,24,9848022334,trivendram
008,Bharathi,Nambiayar,24,9848022333,Chennai
And we have loaded this file into Apache Pig with the relation name student_details as shown
below.
grunt> student_details = LOAD 'hdfs://localhost:9000/pig_data/student_details.txt' USING PigStorage(',')
as (id:int, firstname:chararray, lastname:chararray, age:int, phone:chararray, city:chararray);

Now, let us group the records/tuples in the relation by age as shown below.
grunt> group_data = GROUP student_details by age;

Verification
Verify the relation group_data using the DUMP operator as shown below.
grunt> Dump group_data;

Output
Then you will get output displaying the contents of the relation named group_data as shown below.
Here you can observe that the resulting schema has two columns −
 One is age, by which we have grouped the relation.
 The other is a bag, which contains the group of tuples, student records with the respective
age.
(21,{(4,Preethi,Agarwal,21,9848022330,Pune),(1,Rajiv,Reddy,21,9848022337,Hydera bad)})
(22,{(3,Rajesh,Khanna,22,9848022339,Delhi),(2,siddarth,Battacharya,22,984802233 8,Kolkata)})
(23,{(6,Archana,Mishra,23,9848022335,Chennai),(5,Trupthi,Mohanthy,23,9848022336 ,Bhuwaneshwar)})
(24,{(8,Bharathi,Nambiayar,24,9848022333,Chennai),(7,Komal,Nayak,24,9848022334, trivendram)})
You can see the schema of the table after grouping the data using the describe command as shown
below.
grunt> Describe group_data;

group_data: {group: int,student_details: {(id: int,firstname: chararray,

lastname: chararray,age: int,phone: chararray,city: chararray)}}
In the same way, you can get the sample illustration of the schema using the illustrate command as
shown below.
$ Illustrate group_data;

It will produce the following output −

-------------------------------------------------------------------------------------------------
|group_data| group:int |
student_details:bag{:tuple(id:int,firstname:chararray,lastname:chararray,age:int,phone:chararray,city:chararray)
}|
-------------------------------------------------------------------------------------------------
| | 21 | { 4, Preethi, Agarwal, 21, 9848022330, Pune), (1, Rajiv, Reddy, 21, 9848022337,
Hyderabad)}|
| | 2 | {(2,siddarth,Battacharya,22,9848022338,Kolkata),(003,Rajesh,Khanna,22,9848022339,Delhi)}|
-------------------------------------------------------------------------------------------------

Grouping by Multiple Columns

Let us group the relation by age and city as shown below.
grunt> group_multiple = GROUP student_details by (age, city);
You can verify the content of the relation named group_multiple using the Dump operator as shown
below.
grunt> Dump group_multiple;

((21,Pune),{(4,Preethi,Agarwal,21,9848022330,Pune)})
((21,Hyderabad),{(1,Rajiv,Reddy,21,9848022337,Hyderabad)})
((22,Delhi),{(3,Rajesh,Khanna,22,9848022339,Delhi)})
((22,Kolkata),{(2,siddarth,Battacharya,22,9848022338,Kolkata)})
((23,Chennai),{(6,Archana,Mishra,23,9848022335,Chennai)})
((23,Bhuwaneshwar),{(5,Trupthi,Mohanthy,23,9848022336,Bhuwaneshwar)})
((24,Chennai),{(8,Bharathi,Nambiayar,24,9848022333,Chennai)})
(24,trivendram),{(7,Komal,Nayak,24,9848022334,trivendram)})

Group All
You can group a relation by all the columns as shown below.
grunt> group_all = GROUP student_details All;
Now, verify the content of the relation group_all as shown below.
grunt> Dump group_all;

(all,{(8,Bharathi,Nambiayar,24,9848022333,Chennai),(7,Komal,Nayak,24,9848022334 ,trivendram),
(6,Archana,Mishra,23,9848022335,Chennai),(5,Trupthi,Mohanthy,23,9848022336,Bhuw aneshwar),
(4,Preethi,Agarwal,21,9848022330,Pune),(3,Rajesh,Khanna,22,9848022339,Delhi),
(2,siddarth,Battacharya,22,9848022338,Kolkata),(1,Rajiv,Reddy,21,9848022337,Hyd erabad)}

Apache Pig - Cogroup Operator

The COGROUP operator works more or less in the same way as the GROUP operator. The only
difference between the two operators is that the group operator is normally used with one relation,
while the cogroup operator is used in statements involving two or more relations.

Grouping Two Relations using Cogroup

Assume that we have two files namely student_details.txt and employee_details.txt in the HDFS
directory /pig_data/ as shown below.
student_details.txt
001,Rajiv,Reddy,21,9848022337,Hyderabad
002,siddarth,Battacharya,22,9848022338,Kolkata
003,Rajesh,Khanna,22,9848022339,Delhi
004,Preethi,Agarwal,21,9848022330,Pune
005,Trupthi,Mohanthy,23,9848022336,Bhuwaneshwar
006,Archana,Mishra,23,9848022335,Chennai
007,Komal,Nayak,24,9848022334,trivendram
008,Bharathi,Nambiayar,24,9848022333,Chennai
employee_details.txt
001,Robin,22,newyork
002,BOB,23,Kolkata
003,Maya,23,Tokyo
004,Sara,25,London
005,David,23,Bhuwaneshwar
006,Maggy,22,Chennai
And we have loaded these files into Pig with the relation
names student_details and employee_details respectively, as shown below.
grunt> student_details = LOAD 'hdfs://localhost:9000/pig_data/student_details.txt' USING PigStorage(',')
as (id:int, firstname:chararray, lastname:chararray, age:int, phone:chararray, city:chararray);

grunt> employee_details = LOAD 'hdfs://localhost:9000/pig_data/employee_details.txt' USING PigStorage(',')

as (id:int, name:chararray, age:int, city:chararray);

Now, let us group the records/tuples of the relations student_details and employee_details with the
key age, as shown below.
grunt> cogroup_data = COGROUP student_details by age, employee_details by age;

Verification

Verify the relation cogroup_data using the DUMP operator as shown below.
grunt> Dump cogroup_data;
Output

It will produce the following output, displaying the contents of the relation named cogroup_data as
shown below.
(21,{(4,Preethi,Agarwal,21,9848022330,Pune), (1,Rajiv,Reddy,21,9848022337,Hyderabad)},
{ })
(22,{ (3,Rajesh,Khanna,22,9848022339,Delhi), (2,siddarth,Battacharya,22,9848022338,Kolkata) },
{ (6,Maggy,22,Chennai),(1,Robin,22,newyork) })
(23,{(6,Archana,Mishra,23,9848022335,Chennai),(5,Trupthi,Mohanthy,23,9848022336 ,Bhuwaneshwar)},
{(5,David,23,Bhuwaneshwar),(3,Maya,23,Tokyo),(2,BOB,23,Kolkata)})
(24,{(8,Bharathi,Nambiayar,24,9848022333,Chennai),(7,Komal,Nayak,24,9848022334, trivendram)},
{ })
(25,{ },
{(4,Sara,25,London)})
The cogroup operator groups the tuples from each relation according to age where each group
depicts a particular age value.
For example, if we consider the 1st tuple of the result, it is grouped by age 21. And it contains two
bags −
 the first bag holds all the tuples from the first relation (student_details in this case) having
age 21, and
 the second bag contains all the tuples from the second relation (employee_details in this
case) having age 21.
In case a relation doesn‟t have tuples having the age value 21, it returns an empty bag.

Apache Pig - Join Operator

The JOIN operator is used to combine records from two or more relations. While performing a join
operation, we declare one (or a group of) tuple(s) from each relation, as keys. When these keys
match, the two particular tuples are matched, else the records are dropped. Joins can be of the
following types −

 Self-join
 Inner-join
 Outer-join − left join, right join, and full join
This chapter explains with examples how to use the join operator in Pig Latin. Assume that we have
two files namely customers.txt and orders.txt in the /pig_data/ directory of HDFS as shown below.
customers.txt
1,Ramesh,32,Ahmedabad,2000.00
2,Khilan,25,Delhi,1500.00
3,kaushik,23,Kota,2000.00
4,Chaitali,25,Mumbai,6500.00
5,Hardik,27,Bhopal,8500.00
6,Komal,22,MP,4500.00
7,Muffy,24,Indore,10000.00
orders.txt
102,2009-10-08 00:00:00,3,3000
100,2009-10-08 00:00:00,3,1500
101,2009-11-20 00:00:00,2,1560
103,2008-05-20 00:00:00,4,2060
And we have loaded these two files into Pig with the relations customers and orders as shown
below.
grunt> customers = LOAD 'hdfs://localhost:9000/pig_data/customers.txt' USING PigStorage(',')
as (id:int, name:chararray, age:int, address:chararray, salary:int);

grunt> orders = LOAD 'hdfs://localhost:9000/pig_data/orders.txt' USING PigStorage(',')

as (oid:int, date:chararray, customer_id:int, amount:int);

Let us now perform various Join operations on these two relations.

Self - join
Self-join is used to join a table with itself as if the table were two relations, temporarily renaming at
least one relation.
Generally, in Apache Pig, to perform self-join, we will load the same data multiple times, under
different aliases (names). Therefore let us load the contents of the file customers.txt as two tables as
shown below.
grunt> customers1 = LOAD 'hdfs://localhost:9000/pig_data/customers.txt' USING PigStorage(',')
as (id:int, name:chararray, age:int, address:chararray, salary:int);

grunt> customers2 = LOAD 'hdfs://localhost:9000/pig_data/customers.txt' USING PigStorage(',')

as (id:int, name:chararray, age:int, address:chararray, salary:int);

Syntax

Given below is the syntax of performing self-join operation using the JOIN operator.
grunt> Relation3_name = JOIN Relation1_name BY key, Relation2_name BY key ;

Example

Let us perform self-join operation on the relation customers, by joining the two
relations customers1 and customers2 as shown below.
grunt> customers3 = JOIN customers1 BY id, customers2 BY id;

Verification

Verify the relation customers3 using the DUMP operator as shown below.
grunt> Dump customers3;

Output

It will produce the following output, displaying the contents of the relation customers.
(1,Ramesh,32,Ahmedabad,2000,1,Ramesh,32,Ahmedabad,2000)
(2,Khilan,25,Delhi,1500,2,Khilan,25,Delhi,1500)
(3,kaushik,23,Kota,2000,3,kaushik,23,Kota,2000)
(4,Chaitali,25,Mumbai,6500,4,Chaitali,25,Mumbai,6500)
(5,Hardik,27,Bhopal,8500,5,Hardik,27,Bhopal,8500)
(6,Komal,22,MP,4500,6,Komal,22,MP,4500)
(7,Muffy,24,Indore,10000,7,Muffy,24,Indore,10000)
Inner Join
Inner Join is used quite frequently; it is also referred to as equijoin. An inner join returns rows
when there is a match in both tables.
It creates a new relation by combining column values of two relations (say A and B) based upon the
join-predicate. The query compares each row of A with each row of B to find all pairs of rows which
satisfy the join-predicate. When the join-predicate is satisfied, the column values for each matched
pair of rows of A and B are combined into a result row.

Syntax

Here is the syntax of performing inner join operation using the JOIN operator.
grunt> result = JOIN relation1 BY columnname, relation2 BY columnname;

Example

Let us perform inner join operation on the two relations customers and orders as shown below.
grunt> coustomer_orders = JOIN customers BY id, orders BY customer_id;

Verification

Verify the relation coustomer_orders using the DUMP operator as shown below.
grunt> Dump coustomer_orders;

Output

You will get the following output that will the contents of the relation named coustomer_orders.
(2,Khilan,25,Delhi,1500,101,2009-11-20 00:00:00,2,1560)
(3,kaushik,23,Kota,2000,100,2009-10-08 00:00:00,3,1500)
(3,kaushik,23,Kota,2000,102,2009-10-08 00:00:00,3,3000)
(4,Chaitali,25,Mumbai,6500,103,2008-05-20 00:00:00,4,2060)
Note −
Outer Join: Unlike inner join, outer join returns all the rows from at least one of the relations. An
outer join operation is carried out in three ways −

 Left outer join

 Right outer join
 Full outer join

Left Outer Join

The left outer Join operation returns all rows from the left table, even if there are no matches in the
right relation.

Syntax

Given below is the syntax of performing left outer join operation using the JOIN operator.
grunt> Relation3_name = JOIN Relation1_name BY id LEFT OUTER, Relation2_name BY customer_id;

Example

Let us perform left outer join operation on the two relations customers and orders as shown below.
grunt> outer_left = JOIN customers BY id LEFT OUTER, orders BY customer_id;

Verification

Verify the relation outer_left using the DUMP operator as shown below.
grunt> Dump outer_left;

Output

It will produce the following output, displaying the contents of the relation outer_left.
(1,Ramesh,32,Ahmedabad,2000,,,,)
(2,Khilan,25,Delhi,1500,101,2009-11-20 00:00:00,2,1560)
(3,kaushik,23,Kota,2000,100,2009-10-08 00:00:00,3,1500)
(3,kaushik,23,Kota,2000,102,2009-10-08 00:00:00,3,3000)
(4,Chaitali,25,Mumbai,6500,103,2008-05-20 00:00:00,4,2060)
(5,Hardik,27,Bhopal,8500,,,,)
(6,Komal,22,MP,4500,,,,)
(7,Muffy,24,Indore,10000,,,,)

Right Outer Join

The right outer join operation returns all rows from the right table, even if there are no matches in
the left table.

Syntax

Given below is the syntax of performing right outer join operation using the JOIN operator.
grunt> outer_right = JOIN customers BY id RIGHT, orders BY customer_id;

Example

Let us perform right outer join operation on the two relations customers and orders as shown
below.
grunt> outer_right = JOIN customers BY id RIGHT, orders BY customer_id;

Verification

Verify the relation outer_right using the DUMP operator as shown below.
grunt> Dump outer_right

Output

It will produce the following output, displaying the contents of the relation outer_right.
(2,Khilan,25,Delhi,1500,101,2009-11-20 00:00:00,2,1560)
(3,kaushik,23,Kota,2000,100,2009-10-08 00:00:00,3,1500)
(3,kaushik,23,Kota,2000,102,2009-10-08 00:00:00,3,3000)
(4,Chaitali,25,Mumbai,6500,103,2008-05-20 00:00:00,4,2060)

Full Outer Join

The full outer join operation returns rows when there is a match in one of the relations.

Syntax

Given below is the syntax of performing full outer join using the JOIN operator.
grunt> outer_full = JOIN customers BY id FULL OUTER, orders BY customer_id;

Example

Let us perform full outer join operation on the two relations customers and orders as shown below.
grunt> outer_full = JOIN customers BY id FULL OUTER, orders BY customer_id;

Verification

Verify the relation outer_full using the DUMP operator as shown below.
grun> Dump outer_full;

Output

It will produce the following output, displaying the contents of the relation outer_full.
(1,Ramesh,32,Ahmedabad,2000,,,,)
(2,Khilan,25,Delhi,1500,101,2009-11-20 00:00:00,2,1560)
(3,kaushik,23,Kota,2000,100,2009-10-08 00:00:00,3,1500)
(3,kaushik,23,Kota,2000,102,2009-10-08 00:00:00,3,3000)
(4,Chaitali,25,Mumbai,6500,103,2008-05-20 00:00:00,4,2060)
(5,Hardik,27,Bhopal,8500,,,,)
(6,Komal,22,MP,4500,,,,)
(7,Muffy,24,Indore,10000,,,,)

Using Multiple Keys

We can perform JOIN operation using multiple keys.

Syntax

Here is how you can perform a JOIN operation on two tables using multiple keys.
grunt> Relation3_name = JOIN Relation2_name BY (key1, key2), Relation3_name BY (key1, key2);
Assume that we have two files namely employee.txt and employee_contact.txt in
the /pig_data/ directory of HDFS as shown below.
employee.txt
001,Rajiv,Reddy,21,programmer,003
002,siddarth,Battacharya,22,programmer,003
003,Rajesh,Khanna,22,programmer,003
004,Preethi,Agarwal,21,programmer,003
005,Trupthi,Mohanthy,23,programmer,003
006,Archana,Mishra,23,programmer,003
007,Komal,Nayak,24,teamlead,002
008,Bharathi,Nambiayar,24,manager,001
employee_contact.txt
001,9848022337,Rajiv@gmail.com,Hyderabad,003
002,9848022338,siddarth@gmail.com,Kolkata,003
003,9848022339,Rajesh@gmail.com,Delhi,003
004,9848022330,Preethi@gmail.com,Pune,003
005,9848022336,Trupthi@gmail.com,Bhuwaneshwar,003
006,9848022335,Archana@gmail.com,Chennai,003
007,9848022334,Komal@gmail.com,trivendram,002
008,9848022333,Bharathi@gmail.com,Chennai,001
And we have loaded these two files into Pig with relations employee and employee_contact as
shown below.
grunt> employee = LOAD 'hdfs://localhost:9000/pig_data/employee.txt' USING PigStorage(',')
as (id:int, firstname:chararray, lastname:chararray, age:int, designation:chararray, jobid:int);

grunt> employee_contact = LOAD 'hdfs://localhost:9000/pig_data/employee_contact.txt' USING PigStorage(',')

as (id:int, phone:chararray, email:chararray, city:chararray, jobid:int);

Now, let us join the contents of these two relations using the JOIN operator as shown below.
grunt> emp = JOIN employee BY (id,jobid), employee_contact BY (id,jobid);

Verification

Verify the relation emp using the DUMP operator as shown below.
grunt> Dump emp;

Output

It will produce the following output, displaying the contents of the relation named emp as shown
below.
(1,Rajiv,Reddy,21,programmer,113,1,9848022337,Rajiv@gmail.com,Hyderabad,113)
(2,siddarth,Battacharya,22,programmer,113,2,9848022338,siddarth@gmail.com,Kolka ta,113)
(3,Rajesh,Khanna,22,programmer,113,3,9848022339,Rajesh@gmail.com,Delhi,113)
(4,Preethi,Agarwal,21,programmer,113,4,9848022330,Preethi@gmail.com,Pune,113)
(5,Trupthi,Mohanthy,23,programmer,113,5,9848022336,Trupthi@gmail.com,Bhuwaneshw ar,113)
(6,Archana,Mishra,23,programmer,113,6,9848022335,Archana@gmail.com,Chennai,113)
(7,Komal,Nayak,24,teamlead,112,7,9848022334,Komal@gmail.com,trivendram,112)
(8,Bharathi,Nambiayar,24,manager,111,8,9848022333,Bharathi@gmail.com,Chennai,111)

Apache Pig - Cross Operator

The CROSS operator computes the cross-product of two or more relations. This chapter explains
with example how to use the cross operator in Pig Latin.

Syntax
Given below is the syntax of the CROSS operator.
grunt> Relation3_name = CROSS Relation1_name, Relation2_name;

Example
Assume that we have two files namely customers.txt and orders.txt in the /pig_data/ directory of
HDFS as shown below.
customers.txt
1,Ramesh,32,Ahmedabad,2000.00
2,Khilan,25,Delhi,1500.00
3,kaushik,23,Kota,2000.00
4,Chaitali,25,Mumbai,6500.00
5,Hardik,27,Bhopal,8500.00
6,Komal,22,MP,4500.00
7,Muffy,24,Indore,10000.00
orders.txt
102,2009-10-08 00:00:00,3,3000
100,2009-10-08 00:00:00,3,1500
101,2009-11-20 00:00:00,2,1560
103,2008-05-20 00:00:00,4,2060
And we have loaded these two files into Pig with the relations customers and orders as shown
below.
grunt> customers = LOAD 'hdfs://localhost:9000/pig_data/customers.txt' USING PigStorage(',')
as (id:int, name:chararray, age:int, address:chararray, salary:int);

grunt> orders = LOAD 'hdfs://localhost:9000/pig_data/orders.txt' USING PigStorage(',')

as (oid:int, date:chararray, customer_id:int, amount:int);

Let us now get the cross-product of these two relations using the cross operator on these two
relations as shown below.
grunt> cross_data = CROSS customers, orders;

Verification

Verify the relation cross_data using the DUMP operator as shown below.
grunt> Dump cross_data;

Output

It will produce the following output, displaying the contents of the relation cross_data.
(7,Muffy,24,Indore,10000,103,2008-05-20 00:00:00,4,2060)
(7,Muffy,24,Indore,10000,101,2009-11-20 00:00:00,2,1560)
(7,Muffy,24,Indore,10000,100,2009-10-08 00:00:00,3,1500)
(7,Muffy,24,Indore,10000,102,2009-10-08 00:00:00,3,3000)
(6,Komal,22,MP,4500,103,2008-05-20 00:00:00,4,2060)
(6,Komal,22,MP,4500,101,2009-11-20 00:00:00,2,1560)
(6,Komal,22,MP,4500,100,2009-10-08 00:00:00,3,1500)
(6,Komal,22,MP,4500,102,2009-10-08 00:00:00,3,3000)
(5,Hardik,27,Bhopal,8500,103,2008-05-20 00:00:00,4,2060)
(5,Hardik,27,Bhopal,8500,101,2009-11-20 00:00:00,2,1560)
(5,Hardik,27,Bhopal,8500,100,2009-10-08 00:00:00,3,1500)
(5,Hardik,27,Bhopal,8500,102,2009-10-08 00:00:00,3,3000)
(4,Chaitali,25,Mumbai,6500,103,2008-05-20 00:00:00,4,2060)
(4,Chaitali,25,Mumbai,6500,101,2009-20 00:00:00,4,2060)
(2,Khilan,25,Delhi,1500,101,2009-11-20 00:00:00,2,1560)
(2,Khilan,25,Delhi,1500,100,2009-10-08 00:00:00,3,1500)
(2,Khilan,25,Delhi,1500,102,2009-10-08 00:00:00,3,3000)
(1,Ramesh,32,Ahmedabad,2000,103,2008-05-20 00:00:00,4,2060)
(1,Ramesh,32,Ahmedabad,2000,101,2009-11-20 00:00:00,2,1560)
(1,Ramesh,32,Ahmedabad,2000,100,2009-10-08 00:00:00,3,1500)
(1,Ramesh,32,Ahmedabad,2000,102,2009-10-08 00:00:00,3,3000)-11-20 00:00:00,2,1560)
(4,Chaitali,25,Mumbai,6500,100,2009-10-08 00:00:00,3,1500)
(4,Chaitali,25,Mumbai,6500,102,2009-10-08 00:00:00,3,3000)
(3,kaushik,23,Kota,2000,103,2008-05-20 00:00:00,4,2060)
(3,kaushik,23,Kota,2000,101,2009-11-20 00:00:00,2,1560)
(3,kaushik,23,Kota,2000,100,2009-10-08 00:00:00,3,1500)
(3,kaushik,23,Kota,2000,102,2009-10-08 00:00:00,3,3000)
(2,Khilan,25,Delhi,1500,103,2008-05-20 00:00:00,4,2060)
(2,Khilan,25,Delhi,1500,101,2009-11-20 00:00:00,2,1560)
(2,Khilan,25,Delhi,1500,100,2009-10-08 00:00:00,3,1500)
(2,Khilan,25,Delhi,1500,102,2009-10-08 00:00:00,3,3000)
(1,Ramesh,32,Ahmedabad,2000,103,2008-05-20 00:00:00,4,2060)
(1,Ramesh,32,Ahmedabad,2000,101,2009-11-20 00:00:00,2,1560)
(1,Ramesh,32,Ahmedabad,2000,100,2009-10-08 00:00:00,3,1500)
(1,Ramesh,32,Ahmedabad,2000,102,2009-10-08 00:00:00,3,3000)

Apache Pig - Union Operator

The UNION operator of Pig Latin is used to merge the content of two relations. To perform UNION
operation on two relations, their columns and domains must be identical.

Syntax
Given below is the syntax of the UNION operator.
grunt> Relation_name3 = UNION Relation_name1, Relation_name2;

Example
Assume that we have two files namely student_data1.txt and student_data2.txt in
the /pig_data/ directory of HDFS as shown below.
Student_data1.txt
001,Rajiv,Reddy,9848022337,Hyderabad
002,siddarth,Battacharya,9848022338,Kolkata
003,Rajesh,Khanna,9848022339,Delhi
004,Preethi,Agarwal,9848022330,Pune
005,Trupthi,Mohanthy,9848022336,Bhuwaneshwar
006,Archana,Mishra,9848022335,Chennai.
Student_data2.txt
7,Komal,Nayak,9848022334,trivendram.
8,Bharathi,Nambiayar,9848022333,Chennai.
And we have loaded these two files into Pig with the relations student1 and student2 as shown
below.
grunt> student1 = LOAD 'hdfs://localhost:9000/pig_data/student_data1.txt' USING PigStorage(',')
as (id:int, firstname:chararray, lastname:chararray, phone:chararray, city:chararray);

grunt> student2 = LOAD 'hdfs://localhost:9000/pig_data/student_data2.txt' USING PigStorage(',')

as (id:int, firstname:chararray, lastname:chararray, phone:chararray, city:chararray);

Let us now merge the contents of these two relations using the UNION operator as shown below.
grunt> student = UNION student1, student2;

Verification

Verify the relation student using the DUMP operator as shown below.
grunt> Dump student;

Output

It will display the following output, displaying the contents of the relation student.
(1,Rajiv,Reddy,9848022337,Hyderabad) (2,siddarth,Battacharya,9848022338,Kolkata)
(3,Rajesh,Khanna,9848022339,Delhi)
(4,Preethi,Agarwal,9848022330,Pune)
(5,Trupthi,Mohanthy,9848022336,Bhuwaneshwar)
(6,Archana,Mishra,9848022335,Chennai)
(7,Komal,Nayak,9848022334,trivendram)
(8,Bharathi,Nambiayar,9848022333,Chennai)

Apache Pig - Split Operator

The SPLIT operator is used to split a relation into two or more relations.

Syntax
Given below is the syntax of the SPLIT operator.
grunt> SPLIT Relation1_name INTO Relation2_name IF (condition1), Relation2_name (condition2),

Example
Assume that we have a file named student_details.txt in the HDFS directory /pig_data/ as shown
below.
student_details.txt
001,Rajiv,Reddy,21,9848022337,Hyderabad
002,siddarth,Battacharya,22,9848022338,Kolkata
003,Rajesh,Khanna,22,9848022339,Delhi
004,Preethi,Agarwal,21,9848022330,Pune
005,Trupthi,Mohanthy,23,9848022336,Bhuwaneshwar
006,Archana,Mishra,23,9848022335,Chennai
007,Komal,Nayak,24,9848022334,trivendram
008,Bharathi,Nambiayar,24,9848022333,Chennai
And we have loaded this file into Pig with the relation name student_details as shown below.
student_details = LOAD 'hdfs://localhost:9000/pig_data/student_details.txt' USING PigStorage(',')
 as (id:int, firstname:chararray, lastname:chararray, age:int, phone:chararray, city:chararray);

Let us now split the relation into two, one listing the employees of age less than 23, and the other
listing the employees having the age between 22 and 25.
SPLIT student_details into student_details1 if age<23, student_details2 if (22<age and age>25);

Verification

Verify the relations student_details1 and student_details2 using the DUMP operator as shown
below.
grunt> Dump student_details1;

grunt> Dump student_details2;

Output

It will produce the following output, displaying the contents of the

relations student_details1 and student_details2 respectively.
grunt> Dump student_details1;
(1,Rajiv,Reddy,21,9848022337,Hyderabad)
(2,siddarth,Battacharya,22,9848022338,Kolkata)
(3,Rajesh,Khanna,22,9848022339,Delhi)
(4,Preethi,Agarwal,21,9848022330,Pune)

grunt> Dump student_details2;

(5,Trupthi,Mohanthy,23,9848022336,Bhuwaneshwar)
(6,Archana,Mishra,23,9848022335,Chennai)
(7,Komal,Nayak,24,9848022334,trivendram)
(8,Bharathi,Nambiayar,24,9848022333,Chennai)

Apache Pig - Filter Operator

The FILTER operator is used to select the required tuples from a relation based on a condition.

Syntax
Given below is the syntax of the FILTER operator.
grunt> Relation2_name = FILTER Relation1_name BY (condition);

Example
Assume that we have a file named student_details.txt in the HDFS directory /pig_data/ as shown
below.
student_details.txt
001,Rajiv,Reddy,21,9848022337,Hyderabad
002,siddarth,Battacharya,22,9848022338,Kolkata
003,Rajesh,Khanna,22,9848022339,Delhi
004,Preethi,Agarwal,21,9848022330,Pune
005,Trupthi,Mohanthy,23,9848022336,Bhuwaneshwar
006,Archana,Mishra,23,9848022335,Chennai
007,Komal,Nayak,24,9848022334,trivendram
008,Bharathi,Nambiayar,24,9848022333,Chennai
And we have loaded this file into Pig with the relation name student_details as shown below.
grunt> student_details = LOAD 'hdfs://localhost:9000/pig_data/student_details.txt' USING PigStorage(',')
as (id:int, firstname:chararray, lastname:chararray, age:int, phone:chararray, city:chararray);

Let us now use the Filter operator to get the details of the students who belong to the city Chennai.
filter_data = FILTER student_details BY city == 'Chennai';

Verification

Verify the relation filter_data using the DUMP operator as shown below.
grunt> Dump filter_data;

Output

It will produce the following output, displaying the contents of the relation filter_data as follows.
(6,Archana,Mishra,23,9848022335,Chennai)
(8,Bharathi,Nambiayar,24,9848022333,Chennai)

Apache Pig - Distinct Operator

The DISTINCT operator is used to remove redundant (duplicate) tuples from a relation.

Syntax
Given below is the syntax of the DISTINCT operator.
grunt> Relation_name2 = DISTINCT Relatin_name1;

Example
Assume that we have a file named student_details.txt in the HDFS directory /pig_data/ as shown
below.
student_details.txt
001,Rajiv,Reddy,9848022337,Hyderabad
002,siddarth,Battacharya,9848022338,Kolkata
002,siddarth,Battacharya,9848022338,Kolkata
003,Rajesh,Khanna,9848022339,Delhi
003,Rajesh,Khanna,9848022339,Delhi
004,Preethi,Agarwal,9848022330,Pune
005,Trupthi,Mohanthy,9848022336,Bhuwaneshwar
006,Archana,Mishra,9848022335,Chennai
006,Archana,Mishra,9848022335,Chennai
And we have loaded this file into Pig with the relation name student_details as shown below.
grunt> student_details = LOAD 'hdfs://localhost:9000/pig_data/student_details.txt' USING PigStorage(',')
as (id:int, firstname:chararray, lastname:chararray, phone:chararray, city:chararray);

Let us now remove the redundant (duplicate) tuples from the relation named student_details using
the DISTINCT operator, and store it as another relation named distinct_data as shown below.
grunt> distinct_data = DISTINCT student_details;

Verification

Verify the relation distinct_data using the DUMP operator as shown below.
grunt> Dump distinct_data;

Output

It will produce the following output, displaying the contents of the relation distinct_data as follows.
(1,Rajiv,Reddy,9848022337,Hyderabad)
(2,siddarth,Battacharya,9848022338,Kolkata)
(3,Rajesh,Khanna,9848022339,Delhi)
(4,Preethi,Agarwal,9848022330,Pune)
(5,Trupthi,Mohanthy,9848022336,Bhuwaneshwar)
(6,Archana,Mishra,9848022335,Chennai)

Apache Pig - Foreach Operator

The FOREACH operator is used to generate specified data transformations based on the column
data.

Syntax
Given below is the syntax of FOREACH operator.
grunt> Relation_name2 = FOREACH Relatin_name1 GENERATE (required data);

Example
Assume that we have a file named student_details.txt in the HDFS directory /pig_data/ as shown
below.
student_details.txt
001,Rajiv,Reddy,21,9848022337,Hyderabad
002,siddarth,Battacharya,22,9848022338,Kolkata
003,Rajesh,Khanna,22,9848022339,Delhi
004,Preethi,Agarwal,21,9848022330,Pune
005,Trupthi,Mohanthy,23,9848022336,Bhuwaneshwar
006,Archana,Mishra,23,9848022335,Chennai
007,Komal,Nayak,24,9848022334,trivendram
008,Bharathi,Nambiayar,24,9848022333,Chennai
And we have loaded this file into Pig with the relation name student_details as shown below.
grunt> student_details = LOAD 'hdfs://localhost:9000/pig_data/student_details.txt' USING PigStorage(',')
as (id:int, firstname:chararray, lastname:chararray,age:int, phone:chararray, city:chararray);

Let us now get the id, age, and city values of each student from the relation student_details and
store it into another relation named foreach_data using the foreach operator as shown below.
grunt> foreach_data = FOREACH student_details GENERATE id,age,city;
Verification

Verify the relation foreach_data using the DUMP operator as shown below.
grunt> Dump foreach_data;

Output

It will produce the following output, displaying the contents of the relation foreach_data.
(1,21,Hyderabad)
(2,22,Kolkata)
(3,22,Delhi)
(4,21,Pune)
(5,23,Bhuwaneshwar)
(6,23,Chennai)
(7,24,trivendram)
(8,24,Chennai)

Apache Pig - Order By

The ORDER BY operator is used to display the contents of a relation in a sorted order based on one or more fields.

Syntax
Given below is the syntax of the ORDER BY operator.
grunt> Relation_name2 = ORDER Relatin_name1 BY (ASC|DESC);

Example
Assume that we have a file named student_details.txt in the HDFS directory /pig_data/ as shown below.

student_details.txt
001,Rajiv,Reddy,21,9848022337,Hyderabad
002,siddarth,Battacharya,22,9848022338,Kolkata
003,Rajesh,Khanna,22,9848022339,Delhi
004,Preethi,Agarwal,21,9848022330,Pune
005,Trupthi,Mohanthy,23,9848022336,Bhuwaneshwar
006,Archana,Mishra,23,9848022335,Chennai
007,Komal,Nayak,24,9848022334,trivendram
008,Bharathi,Nambiayar,24,9848022333,Chennai
And we have loaded this file into Pig with the relation name student_details as shown below.

grunt> student_details = LOAD 'hdfs://localhost:9000/pig_data/student_details.txt' USING PigStorage(',')

as (id:int, firstname:chararray, lastname:chararray,age:int, phone:chararray, city:chararray);

Let us now sort the relation in a descending order based on the age of the student and store it into another relation
named order_by_data using the ORDER BY operator as shown below.

grunt> order_by_data = ORDER student_details BY age DESC;

Verification

Verify the relation order_by_data using the DUMP operator as shown below.

grunt> Dump order_by_data;

Output
It will produce the following output, displaying the contents of the relation order_by_data.
(8,Bharathi,Nambiayar,24,9848022333,Chennai)
(7,Komal,Nayak,24,9848022334,trivendram)
(6,Archana,Mishra,23,9848022335,Chennai)
(5,Trupthi,Mohanthy,23,9848022336,Bhuwaneshwar)
(3,Rajesh,Khanna,22,9848022339,Delhi)
(2,siddarth,Battacharya,22,9848022338,Kolkata)
(4,Preethi,Agarwal,21,9848022330,Pune)
(1,Rajiv,Reddy,21,9848022337,Hyderabad)

The LIMIT operator is used to get a limited number of tuples from a relation.

Syntax
Given below is the syntax of the LIMIT operator.
grunt> Result = LIMIT Relation_name required number of tuples;

Example
Assume that we have a file named student_details.txt in the HDFS directory /pig_data/ as shown below.
student_details.txt
001,Rajiv,Reddy,21,9848022337,Hyderabad
002,siddarth,Battacharya,22,9848022338,Kolkata
003,Rajesh,Khanna,22,9848022339,Delhi
004,Preethi,Agarwal,21,9848022330,Pune
005,Trupthi,Mohanthy,23,9848022336,Bhuwaneshwar
006,Archana,Mishra,23,9848022335,Chennai
007,Komal,Nayak,24,9848022334,trivendram
008,Bharathi,Nambiayar,24,9848022333,Chennai
And we have loaded this file into Pig with the relation name student_details as shown below.

grunt> student_details = LOAD 'hdfs://localhost:9000/pig_data/student_details.txt' USING PigStorage(',')

as (id:int, firstname:chararray, lastname:chararray,age:int, phone:chararray, city:chararray);

Now, let‟s sort the relation in descending order based on the age of the student and store it into another relation
named limit_data using the ORDER BY operator as shown below.

grunt> limit_data = LIMIT student_details 4;

Verification

Verify the relation limit_data using the DUMP operator as shown below.

grunt> Dump limit_data;

Output

It will produce the following output, displaying the contents of the relation limit_data as follows.
(1,Rajiv,Reddy,21,9848022337,Hyderabad)
(2,siddarth,Battacharya,22,9848022338,Kolkata)
(3,Rajesh,Khanna,22,9848022339,Delhi)
(4,Preethi,Agarwal,21,9848022330,Pune)

Apache Pig - Eval Functions

Apache Pig provides various built-in functions namely eval, load, store, math, string, bag and tuple functions.
Eval Functions
Given below is the list of eval functions provided by Apache Pig.

S.N. Function & Description

1 AVG()

To compute the average of the numerical values within a bag.

2 BagToString()

To concatenate the elements of a bag into a string. While concatenating, we can place a
delimiter between these values (optional).

3 CONCAT()

To concatenate two or more expressions of same type.

4 COUNT()

To get the number of elements in a bag, while counting the number of tuples in a bag.

5 COUNT_STAR()

It is similar to the COUNT() function. It is used to get the number of elements in a bag.

6 DIFF()

To compare two bags (fields) in a tuple.

7 IsEmpty()

To check if a bag or map is empty.

8 MAX()

To calculate the highest value for a column (numeric values or chararrays) in a single-
column bag.

9 MIN()

To get the minimum (lowest) value (numeric or chararray) for a certain column in a single-
column bag.

10 PluckTuple()

Using the Pig Latin PluckTuple() function, we can define a string Prefix and filter the
columns in a relation that begin with the given prefix.

11 SIZE()

To compute the number of elements based on any Pig data type.

12 SUBTRACT()

To subtract two bags. It takes two bags as inputs and returns a bag which contains the tuples
of the first bag that are not in the second bag.
13 SUM()

To get the total of the numeric values of a column in a single-column bag.

14 TOKENIZE()

To split a string (which contains a group of words) in a single tuple and return a bag which
contains the output of the split operation.

Apache Pig - AVG()

The Pig-Latin AVG() function is used to compute the average of the numerical values within a bag. While calculating the average
value, the AVG() function ignores the NULL values.
Note −

 To get the global average value, we need to perform a Group All operation, and calculate the average value using
the AVG() function.

 To get the average value of a group, we need to group it using the Group By operator and proceed with the average
function.

Syntax
Given below is the syntax of the AVG() function.
grunt> AVG(expression)

Example
Assume that we have a file named student_details.txt in the HDFS directory /pig_data/ as shown below.

student_details.txt
001,Rajiv,Reddy,21,9848022337,Hyderabad,89
002,siddarth,Battacharya,22,9848022338,Kolkata,78
003,Rajesh,Khanna,22,9848022339,Delhi,90
004,Preethi,Agarwal,21,9848022330,Pune,93
005,Trupthi,Mohanthy,23,9848022336,Bhuwaneshwar,75
006,Archana,Mishra,23,9848022335,Chennai,87
007,Komal,Nayak,24,9848022334,trivendram,83
008,Bharathi,Nambiayar,24,9848022333,Chennai,72
And we have loaded this file into Pig with the relation name student_details as shown below.

grunt> student_details = LOAD 'hdfs://localhost:9000/pig_data/student_details.txt' USING PigStorage(',')

as (id:int, firstname:chararray, lastname:chararray, age:int, phone:chararray, city:chararray, gpa:int);

Calculating the Average GPA

We can use the built-in function AVG() (case-sensitive) to calculate the average of a set of numerical values. Let‟s group the
relation student_details using the Group All operator, and store the result in the relation named student_group_all as shown
below.

grunt> student_group_all = Group student_details All;

This will produce a relation as shown below.

grunt> Dump student_group_all;

(all,{(8,Bharathi,Nambiayar,24,9848022333,Chennai,72),
(7,Komal,Nayak,24,9848022 334,trivendram,83),
(6,Archana,Mishra,23,9848022335,Chennai,87),
(5,Trupthi,Mohan thy,23,9848022336,Bhuwaneshwar,75),
(4,Preethi,Agarwal,21,9848022330,Pune,93),
(3 ,Rajesh,Khanna,22,9848022339,Delhi,90),
(2,siddarth,Battacharya,22,9848022338,Ko lkata,78),
(1,Rajiv,Reddy,21,9848022337,Hyderabad,89)})
Let us now calculate the global average GPA of all the students using the AVG() function as shown below.

grunt> student_gpa_avg = foreach student_group_all Generate

(student_details.firstname, student_details.gpa), AVG(student_details.gpa);

Verification

Verify the relation student_gpa_avg using the DUMP operator as shown below.

grunt> Dump student_gpa_avg;

Output

It will display the contents of the relation student_gpa_avg as follows.

(({(Bharathi),(Komal),(Archana),(Trupthi),(Preethi),(Rajesh),(siddarth),(Rajiv) },
{ (72) , (83) , (87) , (75) , (93) , (90) , (78) , (89) })

Apache Pig - BagToString()

The Pig Latin BagToString() function is used to concatenate the elements of a bag into a string. While concatenating, we can
place a delimiter between these values (optional).

Generally bags are disordered and can be arranged by using ORDER BY operator.

Syntax
Given below is the syntax of the BagToString() function.
grunt> BagToString(vals:bag [, delimiter:chararray])

Example
Assume that we have a file named dateofbirth.txt in the HDFS directory /pig_data/ as shown below. This file contains the date-
of-births.

dateofbirth.txt
22,3,1990
23,11,1989
1,3,1998
2,6,1980
26,9,1989
And we have loaded this file into Pig with the relation name dob as shown below.

grunt> dob = LOAD 'hdfs://localhost:9000/pig_data/dateofbirth.txt' USING PigStorage(',')

as (day:int, month:int, year:int);

Converting Bag to String

Using the bagtostring() function, we can convert the data in the bag to string. Let us group the dob relation. The group operation
will produce a bag containing all the tuples of the relation.
Group the relation dob using the Group All operator, and store the result in the relation named group_dob as shown below.

grunt> group_dob = Group dob All;

It will produce a relation as shown below.

grunt> Dump group_dob;

(all,{(26,9,1989),(2,6,1980),(1,3,1998),(23,11,1989),(22,3,1990)})
Here, we can observe a bag having all the date-of-births as tuples of it. Now, let‟s convert the bag to string using the
function BagToString().

grunt> dob_string = foreach group_dob Generate BagToString(dob);

Verification

Verify the relation dob_string using the DUMP operator as shown below.

grunt> Dump dob_string;

Output

It will produce the following output, displaying the contents of the relation dob_string.
(26_9_1989_2_6_1980_1_3_1998_23_11_1989_22_3_1990)

Apache Pig - CONCAT()

The CONCAT() function of Pig Latin is used to concatenate two or more expressions of the same type.

Syntax
grunt> CONCAT (expression, expression, [...expression])

Example
Assume that we have a file named student_details.txt in the HDFS directory /pig_data/ as shown below.

student_details.txt
001,Rajiv,Reddy,21,9848022337,Hyderabad,89
002,siddarth,Battacharya,22,9848022338,Kolkata,78
003,Rajesh,Khanna,22,9848022339,Delhi,90
004,Preethi,Agarwal,21,9848022330,Pune,93
005,Trupthi,Mohanthy,23,9848022336,Bhuwaneshwar,75
006,Archana,Mishra,23,9848022335,Chennai,87
007,Komal,Nayak,24,9848022334,trivendram,83
008,Bharathi,Nambiayar,24,9848022333,Chennai,72
And we have loaded this file into Pig with the relation name student_details as shown below.

grunt> student_details = LOAD 'hdfs://localhost:9000/pig_data/student_details.txt' USING PigStorage(',')

as (id:int, firstname:chararray, lastname:chararray, age:int, phone:chararray, city:chararray, gpa:int);

Concatenating Two Strings

We can use the CONCAT() function to concatenate two or more expressions. First of all, verify the contents of
the student_details relation using the Dump operator as shown below.
grunt> Dump student_details;

( 1,Rajiv,Reddy,21,9848022337,Hyderabad,89 )
( 2,siddarth,Battacharya,22,9848022338,Kolkata,78 )
( 3,Rajesh,Khanna,22,9848022339,Delhi,90 )
( 4,Preethi,Agarwal,21,9848022330,Pune,93 )
( 5,Trupthi,Mohanthy,23,9848022336,Bhuwaneshwar,75 )
( 6,Archana,Mishra,23,9848022335,Chennai,87 )
( 7,Komal,Nayak,24,9848022334,trivendram,83 )
( 8,Bharathi,Nambiayar,24,9848022333,Chennai,72 )
And, verify the schema using describe operator as shown below.
grunt> Describe student_details;

student_details: {id: int, firstname: chararray, lastname: chararray, age: int,

phone: chararray, city: chararray, gpa: int}
In the above schema, you can observe that the name of the student is represented using two chararray values
namely firstname and lastname. Let us concatinate these two values using the CONCAT() function.

grunt> student_name_concat = foreach student_details Generate CONCAT (firstname, lastname);

Verification

Verify the relation student_name_concat using the DUMP operator as shown below.

grunt> Dump student_name_concat;

Output

It will produce the following output, displaying the contents of the relation student_name_concat.
(RajivReddy)
(siddarthBattacharya)
(RajeshKhanna)
(PreethiAgarwal)
(TrupthiMohanthy)
(ArchanaMishra)
(KomalNayak)
(BharathiNambiayar)
We can also use an optional delimiter between the two expressions as shown below.

grunt> CONCAT(firstname, '_',lastname);

Now, let us concatenate the first name and last name of the student records in the student_details relation by placing „_‟ between
them as shown below.

grunt> student_name_concat = foreach student_details GENERATE CONCAT(firstname, '_',lastname);

Verification

Verify the relation student_name_concat using the DUMP operator as shown below.

grunt> Dump student_name_concat;

Output

It will produce the following output, displaying the contents of the relation student_name_concat as follows.
(Rajiv_Reddy)
(siddarth_Battacharya)
(Rajesh_Khanna)
(Preethi_Agarwal)
(Trupthi_Mohanthy)
(Archana_Mishra)
(Komal_Nayak)
(Bharathi_Nambiayar)

Apache Pig - COUNT()

The COUNT() function of Pig Latin is used to get the number of elements in a bag. While counting the number of tuples in a bag,
the COUNT() function ignores (will not count) the tuples having a NULL value in the FIRST FIELD.
Note −

 To get the global count value (total number of tuples in a bag), we need to perform a Group All operation, and calculate
the count value using the COUNT() function.

 To get the count value of a group (Number of tuples in a group), we need to group it using the Group By operator and
proceed with the count function.

Syntax
Given below is the syntax of the COUNT() function.
grunt> COUNT(expression)

Example
Assume that we have a file named student_details.txt in the HDFS directory /pig_data/ as shown below.

student_details.txt
001,Rajiv,Reddy,21,9848022337,Hyderabad,89
002,siddarth,Battacharya,22,9848022338,Kolkata,78
003,Rajesh,Khanna,22,9848022339,Delhi,90
004,Preethi,Agarwal,21,9848022330,Pune,93
005,Trupthi,Mohanthy,23,9848022336,Bhuwaneshwar,75
006,Archana,Mishra,23,9848022335,Chennai,87
007,Komal,Nayak,24,9848022334,trivendram,83
008,Bharathi,Nambiayar,24,9848022333,Chennai,72
And we have loaded this file into Pig with the relation named student_details as shown below.

grunt> student_details = LOAD 'hdfs://localhost:9000/pig_data/student_details.txt' USING PigStorage(',')

as (id:int, firstname:chararray, lastname:chararray, age:int, phone:chararray, city:chararray, gpa:int);

Calculating the Number of Tuples

We can use the built-in function COUNT() (case sensitive) to calculate the number of tuples in a relation. Let us group the
relation student_details using the Group All operator, and store the result in the relation named student_group_all as shown
below.

grunt> student_group_all = Group student_details All;

It will produce a relation as shown below.

grunt> Dump student_group_all;

(all,{(8,Bharathi,Nambiayar,24,9848022333,Chennai,72),
(7,Komal,Nayak,24,9848022 334,trivendram,83),
(6,Archana,Mishra,23,9848022335,Chennai,87),
(5,Trupthi,Mohan thy,23,9848022336,Bhuwaneshwar,75),
(4,Preethi,Agarwal,21,9848022330,Pune,93),
(3 ,Rajesh,Khanna,22,9848022339,Delhi,90),
(2,siddarth,Battacharya,22,9848022338,Ko lkata,78),
(1,Rajiv,Reddy,21,9848022337,Hyderabad,89)})
Let us now calculate number of tuples/records in the relation.

grunt> student_count = foreach student_group_all Generate COUNT(student_details.gpa);

Verification

Verify the relation student_count using the DUMP operator as shown below.

grunt> Dump student_count;

Output

It will produce the following output, displaying the contents of the relation student_count.
8
Apache Pig - COUNT_STAR()
The COUNT_STAR() function of Pig Latin is similar to the COUNT() function. It is used to get the number of elements in a bag.
While counting the elements, the COUNT_STAR() function includes the NULL values.
Note −

 To get the global count value (total number of tuples in a bag), we need to perform a Group All operation, and calculate
the count_star value using the COUNT_STAR() function.
  To get the count value of a group (Number of tuples in a group), we need to group it using the Group By operator and
proceed with the count_star function.

Syntax
Given below is the syntax of the COUNT_STAR() function.
grunt> COUNT_STAR(expression)

Example
Assume that we have a file named student_details.txt in the HDFS directory /pig_data/ as shown below. This file contains an
empty record.

student_details.txt
,,,,,,,
001,Rajiv,Reddy,21,9848022337,Hyderabad,89
002,siddarth,Battacharya,22,9848022338,Kolkata,78
003,Rajesh,Khanna,22,9848022339,Delhi,90
004,Preethi,Agarwal,21,9848022330,Pune,93
005,Trupthi,Mohanthy,23,9848022336,Bhuwaneshwar,75
006,Archana,Mishra,23,9848022335,Chennai,87
007,Komal,Nayak,24,9848022334,trivendram,83
008,Bharathi,Nambiayar,24,9848022333,Chennai,72
And we have loaded this file into Pig with the relation name student_details as shown below.

grunt> student_details = LOAD 'hdfs://localhost:9000/pig_data/student_details.txt' USING PigStorage(',')

as (id:int, firstname:chararray, lastname:chararray, age:int, phone:chararray, city:chararray, gpa:int);

Calculating the Number of Tuples

We can use the built-in function COUNT_STAR() to calculate the number of tuples in a relation. Let us group the
relation student_details using the Group All operator, and store the result in the relation named student_group_all as shown
below.

grunt> student_group_all = Group student_details All;

It will produce a relation as shown below.

grunt> Dump student_group_all;

(all,{(8,Bharathi,Nambiayar,24,9848022333,Chennai,72),
(7,Komal,Nayak,24,9848022 334,trivendram,83),
(6,Archana,Mishra,23,9848022335,Chennai,87),
(5,Trupthi,Mohan thy,23,9848022336,Bhuwaneshwar,75),
(4,Preethi,Agarwal,21,9848022330,Pune,93),
(3 ,Rajesh,Khanna,22,9848022339,Delhi,90),
(2,siddarth,Battacharya,22,9848022338,Ko lkata,78),
(1,Rajiv,Reddy,21,9848022337,Hyderabad,89),
( , , , , , , )})
Let us now calculate the number of tuples/records in the relation.

grunt> student_count = foreach student_group_all Generate COUNT_STAR(student_details.gpa);

Verification

Verify the relation student_count using the DUMP operator as shown below.

grunt> Dump student_count;

Output

It will produce the following output, displaying the contents of the relation student_count.
9
Since we have used the function COUNT_STAR(), it included the null tuple and returned 9.

Apache Pig - DIFF()

The DIFF() function of Pig Latin is used to compare two bags (fields) in a tuple. It takes two fields of a tuple as input and matches
them. If they match, it returns an empty bag. If they do not match, it finds the elements that exist in one field (bag) and not found
in the other, and returns these elements by wrapping them within a bag.

Syntax
Given below is the syntax of the DIFF() function.
grunt> DIFF (expression, expression)

Example
Generally the DIFF() function compares two bags in a tuple. Given below is its example, here we create two relations, cogroup
them, and calculate the difference between them.

Assume that we have two files namely emp_sales.txt and emp_bonus.txt in the HDFS directory /pig_data/ as shown below.
The emp_sales.txt contains the details of the employees of the sales department and the emp_bonus.txt contains the employee
details who got bonus.

emp_sales.txt
1,Robin,22,25000,sales
2,BOB,23,30000,sales
3,Maya,23,25000,sales
4,Sara,25,40000,sales
5,David,23,45000,sales
6,Maggy,22,35000,sales
emp_bonus.txt
1,Robin,22,25000,sales
2,Jaya,23,20000,admin
3,Maya,23,25000,sales
4,Alia,25,50000,admin
5,David,23,45000,sales
6,Omar,30,30000,admin
And we have loaded these files into Pig, with the relation names emp_sales and emp_bonus respectively.

grunt> emp_sales = LOAD 'hdfs://localhost:9000/pig_data/emp_sales.txt' USING PigStorage(',')

as (sno:int, name:chararray, age:int, salary:int, dept:chararray);

grunt> emp_bonus = LOAD 'hdfs://localhost:9000/pig_data/emp_bonus.txt' USING PigStorage(',')

as (sno:int, name:chararray, age:int, salary:int, dept:chararray);

Group the records/tuples of the relations emp_sales and emp_bonus with the key sno, using the COGROUP operator as shown
below.

grunt> cogroup_data = COGROUP emp_sales by sno, emp_bonus by sno;

Verify the relation cogroup_data using the DUMP operator as shown below.
grunt> Dump cogroup_data;

(1,{(1,Robin,22,25000,sales)},{(1,Robin,22,25000,sales)})
(2,{(2,BOB,23,30000,sales)},{(2,Jaya,23,20000,admin)})
(3,{(3,Maya,23,25000,sales)},{(3,Maya,23,25000,sales)})
(4,{(4,Sara,25,40000,sales)},{(4,Alia,25,50000,admin)})
(5,{(5,David,23,45000,sales)},{(5,David,23,45000,sales)})
(6,{(6,Maggy,22,35000,sales)},{(6,Omar,30,30000,admin)})

Calculating the Difference between Two Relations

Let us now calculate the difference between the two relations using DIFF() function and store it in the relation diff_data as shown
below.

grunt> diff_data = FOREACH cogroup_data GENERATE DIFF(emp_sales,emp_bonus);

Verification

Verify the relation diff_data using the DUMP operator as shown below.
grunt> Dump diff_data;

({})
({(2,BOB,23,30000,sales),(2,Jaya,23,20000,admin)})
({})
({(4,Sara,25,40000,sales),(4,Alia,25,50000,admin)})
({})
({(6,Maggy,22,35000,sales),(6,Omar,30,30000,admin)})
The diff_data relation will have an empty tuple if the records in emp_bonus and emp_sales match. In other cases, it will hold
tuples from both the relations (tuples that differ).
For example, if you consider the records having sno as 1, then you will find them same in both the relations
((1,Robin,22,25000,sales), (1,Robin,22,25000,sales)). Therefore, in the diff_data relation, which is the result of DIFF() function,
you will get an empty tuple for sno 1.

Apache Pig - IsEmpty()

The IsEmpty() function of Pig Latin is used to check if a bag or map is empty.

Syntax
Given below is the syntax of the IsEmpty() function.
grunt> IsEmpty(expression)

Example
Assume that we have two files namely emp_sales.txt and emp_bonus.txt in the HDFS directory /pig_data/ as shown below.
The emp_sales.txt contains the details of the employees of the sales department and the emp_bonus.txt contains the employee
details who got bonus.

emp_sales.txt
1,Robin,22,25000,sales
2,BOB,23,30000,sales
3,Maya,23,25000,sales
4,Sara,25,40000,sales
5,David,23,45000,sales
6,Maggy,22,35000,sales
emp_bonus.txt
1,Robin,22,25000,sales
2,Jaya,23,20000,admin
3,Maya,23,25000,sales
4,Alia,25,50000,admin
5,David,23,45000,sales
6,Omar,30,30000,admin
And we have loaded these files into Pig, with the relation names emp_sales and emp_bonus respectively, as shown below.

grunt> emp_sales = LOAD 'hdfs://localhost:9000/pig_data/emp_sales.txt' USING PigStorage(',')

as (sno:int, name:chararray, age:int, salary:int, dept:chararray);

grunt> emp_bonus = LOAD 'hdfs://localhost:9000/pig_data/emp_bonus.txt' USING PigStorage(',')

as (sno:int, name:chararray, age:int, salary:int, dept:chararray);

Let us now group the records/tuples of the relations emp_sales and emp_bonus with the key age, using the cogroup operator as
shown below.

grunt> cogroup_data = COGROUP emp_sales by age, emp_bonus by age;

Verify the relation cogroup_data using the DUMP operator as shown below.
grunt> Dump cogroup_data;

(22,{(6,Maggy,22,35000,sales),(1,Robin,22,25000,sales)}, {(1,Robin,22,25000,sales)})
(23,{(5,David,23,45000,sales),(3,Maya,23,25000,sales),(2,BOB,23,30000,sales)},
 {(5,David,23,45000,sales),(3,Maya,23,25000,sales),(2,Jaya,23,20000,admin)})
(25,{(4,Sara,25,40000,sales)},{(4,Alia,25,50000,admin)})
(30,{},{(6,Omar,30,30000,admin)})
The COGROUP operator groups the tuples from each relation according to age. Each group depicts a particular age value.
For example, if we consider the 1st tuple of the result, it is grouped by age 22. And it contains two bags, the first bag holds all the
tuples from the first relation (student_details in this case) having age 22, and the second bag contains all the tuples from the second
relation (employee_details in this case) having age 22. In case a relation doesn‟t have tuples having the age value 22, it returns an
empty bag.

Getting the Groups having Empty Bags

Let‟s list such empty bags from the emp_sales relation in the group using the IsEmpty() function.

grunt> isempty_data = filter cogroup_data by IsEmpty(emp_sales);

Verification

Verify the relation isempty_data using the DUMP operator as shown below. The emp_sales relation holds the tuples that are not
there in the relation emp_bonus.
grunt> Dump isempty_data;

(30,{},{(6,Omar,30,30000,admin)})

Apache Pig - MAX()

The Pig Latin MAX() function is used to calculate the highest value for a column (numeric values or chararrays) in a single-
column bag. While calculating the maximum value, the Max() function ignores the NULL values.

Note −

 To get the global maximum value, we need to perform a Group All operation, and calculate the maximum value using
the MAX() function.

 To get the maximum value of a group, we need to group it using the Group By operator and proceed with the maximum
function.

Syntax
Given below is the syntax of the Max() function.
grunt> Max(expression)

Example
Assume that we have a file named student_details.txt in the HDFS directory /pig_data/ as shown below.
student_details.txt
001,Rajiv,Reddy,21,9848022337,Hyderabad,89
002,siddarth,Battacharya,22,9848022338,Kolkata,78
003,Rajesh,Khanna,22,9848022339,Delhi,90
004,Preethi,Agarwal,21,9848022330,Pune,93
005,Trupthi,Mohanthy,23,9848022336,Bhuwaneshwar,75
006,Archana,Mishra,23,9848022335,Chennai,87
007,Komal,Nayak,24,9848022334,trivendram,83
008,Bharathi,Nambiayar,24,9848022333,Chennai,72
And we have loaded this file into Pig with the relation name student_details as shown below.

grunt> student_details = LOAD 'hdfs://localhost:9000/pig_data/student_details.txt' USING PigStorage(',')

as (id:int, firstname:chararray, lastname:chararray, age:int, phone:chararray, city:chararray, gpa:int);

Calculating the Maximum GPA

We can use the built-in function MAX() (case-sensitive) to calculate the maximum value from a set of given numerical values. Let
us group the relation student_details using the Group All operator, and store the result in the relation
named student_group_all as shown below.

grunt> student_group_all = Group student_details All;

This will produce a relation as shown below.

grunt> Dump student_group_all;

(all,{(8,Bharathi,Nambiayar,24,9848022333,Chennai,72),
(7,Komal,Nayak,24,9848022 334,trivendram,83),
(6,Archana,Mishra,23,9848022335,Chennai,87),
(5,Trupthi,Mohan thy,23,9848022336,Bhuwaneshwar,75),
(4,Preethi,Agarwal,21,9848022330,Pune,93),
(3,Rajesh,Khanna,22,9848022339,Delhi,90),
(2,siddarth,Battacharya,22,9848022338,Ko lkata,78),
(1,Rajiv,Reddy,21,9848022337,Hyderabad,89)})
Let us now calculate the global maximum of GPA, i.e., maximum among the GPA values of all the students using
the MAX() function as shown below.

grunt> student_gpa_max = foreach student_group_all Generate

(student_details.firstname, student_details.gpa), MAX(student_details.gpa);

Verification

Verify the relation student_gpa_max using the DUMP operator as shown below.

grunt> Dump student_gpa_max;

Output

It will produce the following output, displaying the contents of the relation student_gpa_max.
(({(Bharathi),(Komal),(Archana),(Trupthi),(Preethi),(Rajesh),(siddarth),(Rajiv) } ,
{ (72) , (83) , (87) , (75) , (93) , (90) , (78)

Apache Pig - MIN()

The MIN() function of Pig Latin is used to get the minimum (lowest) value (numeric or chararray) for a certain column in a single-
column bag. While calculating the minimum value, the MIN() function ignores the NULL values.
Note −

 To get the global minimum value, we need to perform a Group All operation, and calculate the minimum value using
the MIN() function.

 To get the minimum value of a group, we need to group it using the Group By operator and proceed with the minimum
function.

Syntax
Given below is the syntax of the MIN() function.
grunt> MIN(expression)

Example
Assume that we have a file named student_details.txt in the HDFS directory /pig_data/ as shown below.

student_details.txt
001,Rajiv,Reddy,21,9848022337,Hyderabad,89
002,siddarth,Battacharya,22,9848022338,Kolkata,78
003,Rajesh,Khanna,22,9848022339,Delhi,90
004,Preethi,Agarwal,21,9848022330,Pune,93
005,Trupthi,Mohanthy,23,9848022336,Bhuwaneshwar,75
006,Archana,Mishra,23,9848022335,Chennai,87
007,Komal,Nayak,24,9848022334,trivendram,83
008,Bharathi,Nambiayar,24,9848022333,Chennai,72
And we have loaded this file into Pig with the relation named student_details as shown below.

grunt> student_details = LOAD 'hdfs://localhost:9000/pig_data/student_details.txt' USING PigStorage(',')

as (id:int, firstname:chararray, lastname:chararray, age:int, phone:chararray, city:chararray, gpa:int);

Calculating the Minimum GPA

We can use the built-in function MIN() (case sensitive) to calculate the minimum value from a set of given numerical values. Let
us group the relation student_details using the Group All operator, and store the result in the relation
named student_group_all as shown below

grunt> student_group_all = Group student_details All;

It will produce a relation as shown below.

grunt> Dump student_group_all;

(all,{(8,Bharathi,Nambiayar,24,9848022333,Chennai,72),
(7,Komal,Nayak,24,9848022 334,trivendram,83),
(6,Archana,Mishra,23,9848022335,Chennai,87),
(5,Trupthi,Mohan thy,23,9848022336,Bhuwaneshwar,75),
(4,Preethi,Agarwal,21,9848022330,Pune,93),
(3 ,Rajesh,Khanna,22,9848022339,Delhi,90),
(2,siddarth,Battacharya,22,9848022338,Ko lkata,78),
(1,Rajiv,Reddy,21,9848022337,Hyderabad,89)})
Let us now calculate the global minimum of GPA, i.e., minimum among the GPA values of all the students using
the MIN() function as shown below.

grunt> student_gpa_min = foreach student_group_all Generate

(student_details.firstname, student_details.gpa), MIN(student_details.gpa);

Verification

Verify the relation student_gpa_min using the DUMP operator as shown below.

grunt> Dump student_gpa_min;

Output

It will produce the following output, displaying the contents of the relation student_gpa_min.
(({(Bharathi),(Komal),(Archana),(Trupthi),(Preethi),(Rajesh),(siddarth),(Rajiv) } ,
{ (72) , (83) , (87) , (75) , (93) , (90) , (78

Apache Pig - PluckTuple()

After performing operations like join to differentiate the columns of the two schemas, we use the function PluckTuple(). To use
this function, first of all, we have to define a string Prefix and we have to filter for the columns in a relation that begin with that
prefix.

Syntax
Given below is the syntax of the PluckTuple() function.
DEFINE pluck PluckTuple(expression1)
DEFINE pluck PluckTuple(expression1,expression3)
pluck(expression2)

Example
Assume that we have two files namely emp_sales.txt and emp_bonus.txt in the HDFS directory /pig_data/.
The emp_sales.txt contains the details of the employees of the sales department and the emp_bonus.txt contains the employee
details who got bonus.
emp_sales.txt
1,Robin,22,25000,sales
2,BOB,23,30000,sales
3,Maya,23,25000,sales
4,Sara,25,40000,sales
5,David,23,45000,sales
6,Maggy,22,35000,sales
emp_bonus.txt
1,Robin,22,25000,sales
2,Jaya,23,20000,admin
3,Maya,23,25000,sales
4,Alia,25,50000,admin
5,David,23,45000,sales
6,Omar,30,30000,admin
And we have loaded these files into Pig, with the relation names emp_sales and emp_bonus respectively.

grunt> emp_sales = LOAD 'hdfs://localhost:9000/pig_data/emp_sales.txt' USING PigStorage(',')

as (sno:int, name:chararray, age:int, salary:int, dept:chararray);

grunt> emp_bonus = LOAD 'hdfs://localhost:9000/pig_data/emp_bonus.txt' USING PigStorage(',')

as (sno:int, name:chararray, age:int, salary:int, dept:chararray);

Join these two relations using the join operator as shown below.

grunt> join_data = join emp_sales by sno, emp_bonus by sno;

Verify the relation join_data using the Dump operator.

grunt> Dump join_data;

(1,Robin,22,25000,sales,1,Robin,22,25000,sales)
(2,BOB,23,30000,sales,2,Jaya,23,20000,admin)
(3,Maya,23,25000,sales,3,Maya,23,25000,sales)
(4,Sara,25,40000,sales,4,Alia,25,50000,admin)
(5,David,23,45000,sales,5,David,23,45000,sales)
(6,Maggy,22,35000,sales,6,Omar,30,30000,admin)

Using PluckTuple() Function

Now, define the required expression by which you want to differentiate the columns using PluckTupe() function.

grunt> DEFINE pluck PluckTuple('a::');

Filter the columns in the join_data relation as shown below.

grunt> data = foreach join_data generate FLATTEN(pluck(*));

Describe the relation named data as shown below.

grunt> Describe data;

data: {emp_sales::sno: int, emp_sales::name: chararray, emp_sales::age: int,

emp_sales::salary: int, emp_sales::dept: chararray, emp_bonus::sno: int,
emp_bonus::name: chararray, emp_bonus::age: int, emp_bonus::salary: int,
emp_bonus::dept: chararray}
Since we have defined the expression as “a::”, the columns of the emp_sales schema are plucked as emp_sales::column
name and the columns of the emp_bonus schema are plucked as emp_bonus::column name

Apache Pig - SIZE()

The SIZE() function of Pig Latin is used to compute the number of elements based on any Pig data type.

Syntax
Given below is the syntax of the SIZE() function.
grunt> SIZE(expression)
The return values vary according to the data types in Apache Pig.

Data type Value

int, long, float, double For all these types, the size function returns 1.

Char array For a char array, the size() function returns the number of characters in
the array.

Byte array For a bytearray, the size() function returns the number of bytes in the
array.

Tuple For a tuple, the size() function returns number of fields in the tuple.

Bag For a bag, the size() function returns number of tuples in the bag.

Map For a map, the size() function returns the number of key/value pairs in
the map.

Example
Assume that we have a file named employee.txt in the HDFS directory /pig_data/ as shown below.
employee.txt
1,John,2007-01-24,250
2,Ram,2007-05-27,220
3,Jack,2007-05-06,170
3,Jack,2007-04-06,100
4,Jill,2007-04-06,220
5,Zara,2007-06-06,300
5,Zara,2007-02-06,350
And we have loaded this file into Pig with the relation name employee_data as shown below.

grunt> employee_data = LOAD 'hdfs://localhost:9000/pig_data/ employee.txt' USING PigStorage(',')

as (id:int, name:chararray, workdate:chararray, daily_typing_pages:int);

Calculating the Size of the Type

To calculate the size of the type of a particular column, we can use the SIZE() function. Let‟s calculate the size of the name type
as shown below.

grunt> size = FOREACH employee_data GENERATE SIZE(name);

Verification

Verify the relation size using the DUMP operator as shown below.

grunt> Dump size;

Output

It will produce the following output, displaying the contents of the relation size as follows. In the example, we have calculated the
size of the name column. Since it is of varchar type, the SIZE() function gives you the number of characters in the name of each
employee.
(4)
(3)
(4)
(4)
(4)
(4)
(4)

Apache Pig - SUBTRACT()

The SUBTRACT() function of Pig Latin is used to subtract two bags. It takes two bags as inputs and returns a bag which contains
the tuples of the first bag that are not in the second bag.

Syntax
Given below is the syntax of the SUBTRACT() function.
grunt> SUBTRACT(expression, expression)

Example
Assume that we have two files namely emp_sales.txt and emp_bonus.txt in the HDFS directory /pig_data/ as shown below.
The emp_sales.txt contains the details of the employees of the sales department and the emp_bonus.txt contains the employee
details who got bonus.
emp_sales.txt
1,Robin,22,25000,sales
2,BOB,23,30000,sales
3,Maya,23,25000,sales
4,Sara,25,40000,sales
5,David,23,45000,sales
6,Maggy,22,35000,sales
emp_bonus.txt
1,Robin,22,25000,sales
2,Jaya,23,20000,admin
3,Maya,23,25000,sales
4,Alia,25,50000,admin
5,David,23,45000,sales
6,Omar,30,30000,admin
And we have loaded these files into Pig, with the relation names emp_sales and emp_bonus respectively.

grunt> emp_sales = LOAD 'hdfs://localhost:9000/pig_data/emp_sales.txt' USING PigStorage(',')

as (sno:int, name:chararray, age:int, salary:int, dept:chararray);

grunt> emp_bonus = LOAD 'hdfs://localhost:9000/pig_data/emp_bonus.txt' USING PigStorage(',')

as (sno:int, name:chararray, age:int, salary:int, dept:chararray);

Let us now group the records/tuples of the relations emp_sales and emp_bonus with the key sno, using the COGROUP operator
as shown below.

grunt> cogroup_data = COGROUP emp_sales by sno, emp_bonus by sno;

Verify the relation cogroup_data using the DUMP operator as shown below.
grunt> Dump cogroup_data;

(1,{(1,Robin,22,25000,sales)},{(1,Robin,22,25000,sales)})
(2,{(2,BOB,23,30000,sales)},{(2,Jaya,23,30000,admin)})
(3,{(3,Maya,23,25000,sales)},{(3,Maya,23,25000,sales)})
(4,{(4,Sara,25,40000,sales)},{(4,Alia,25,50000,admin)})
(5,{(5,David,23,45000,sales)},{(5,David,23,45000,sales)})
(6,{(6,Maggy,22,35000,sales)},{(6,Omar,30,30000,admin)})

Subtracting One Relation from the Other

Let us now subtract the tuples of emp_bonus relation from emp_sales relation. The resulting relation holds the tuples
of emp_sales that are not there in emp_bonus.
grunt> sub_data = FOREACH cogroup_data GENERATE SUBTRACT(emp_sales, emp_bonus);

Verification

Verify the relation sub_data using the DUMP operator as shown below. The emp_sales relation holds the tuples that are not there
in the relation emp_bonus.
grunt> Dump sub_data;

({})
({(2,BOB,23,30000,sales)})
({})
({(4,Sara,25,40000,sales)})
({})
({(6,Maggy,22,35000,sales)})
In the same way, let us subtract the emp_sales relation from emp_bonus relation as shown below.

grunt> sub_data = FOREACH cogroup_data GENERATE SUBTRACT(emp_bonus, emp_sales);

Verify the contents of the sub_data relation using the Dump operator as shown below.
grunt> Dump sub_data;

({})
({(2,Jaya,23,20000,admin)})
({})
({(4,Alia,25,50000,admin)})
({})
({(6,Omar,30,30000,admin)})

Apache Pig - SUM()

You can use the SUM() function of Pig Latin to get the total of the numeric values of a column in a single-column bag. While
computing the total, the SUM() function ignores the NULL values.
Note −

 To get the global sum value, we need to perform a Group All operation, and calculate the sum value using the SUM()
function.

 To get the sum value of a group, we need to group it using the Group By operator and proceed with the sum function.

Syntax
Given below is the syntax of the SUM() function.
grunt> SUM(expression)

Example
Assume that we have a file named employee.txt in the HDFS directory /pig_data/ as shown below.
employee.txt
1,John,2007-01-24,250
2,Ram,2007-05-27,220
3,Jack,2007-05-06,170
3,Jack,2007-04-06,100
4,Jill,2007-04-06,220
5,Zara,2007-06-06,300
5,Zara,2007-02-06,350
And we have loaded this file into Pig with the relation name employee_data as shown below.

grunt> employee_data = LOAD 'hdfs://localhost:9000/pig_data/ employee.txt' USING PigStorage(',')

as (id:int, name:chararray, workdate:chararray, daily_typing_pages:int);

Calculating the Sum of All Daily typing pages

To demonstrate the SUM() function, let‟s try to calculate the total number of pages typed daily of all the employees. We can use
the Apache Pig‟s built-in function SUM() (case sensitive) to calculate the sum of the numerical values. Let us group the relation
employee_data using the Group All operator, and store the result in the relation named employee_group as shown below.

grunt> employee_group = Group employee_data All;

It will produce a relation as shown below.

grunt> Dump employee_group;

(all,{(5,Zara,2007-02-06,350),
(5,Zara,2007-06-06,300),
(4,Jill,2007-0406,220),
(3,Jack,2007-04-06,100),
(3,Jack,2007-05-06,170),
(2,Ram,2007-0527,220),
(1,John,2007-01-24,250)})
Let us now calculate the global sum of the pages typed daily.

grunt> student_workpages_sum = foreach employee_group Generate

(employee_data.name,employee_data.daily_typing_pages),SUM(employee_data.daily_typing_pages);

Verification

Verify the relation student_workpages_sum using the DUMP operator as shown below.

grunt> Dump student_workpages_sum;

Output

It will produce the following output, displaying the contents of the relation student_workpages_sum as follows.
(({ (Zara), (Zara), (Jill) ,(Jack) , (Jack) , (Ram) , (John) },
{ (350) , (300) , (220) ,(100) , (170) , (220) , (250) }),1610)

Apache Pig - TOKENIZE()

The TOKENIZE() function of Pig Latin is used to split a string (which contains a group of words) in a single tuple and returns a
bag which contains the output of the split operation.

Syntax
Given below is the syntax of the TOKENIZE() function.
grunt> TOKENIZE(expression [, 'field_delimiter'])
As a delimeter to the TOKENIZE() function, we can pass space [ ], double quote [" "], coma [ , ], parenthesis [ () ], star [ * ].

Example
Assume that we have a file named student_details.txt in the HDFS directory /pig_data/ as shown below. This file contains the
details of a student like id, name, age and city. If we closely observe, the name of the student includes first and last names
separated by space [ ].
student_details.txt
001,Rajiv Reddy,21,Hyderabad
002,siddarth Battacharya,22,Kolkata
003,Rajesh Khanna,22,Delhi
004,Preethi Agarwal,21,Pune
005,Trupthi Mohanthy,23,Bhuwaneshwar
006,Archana Mishra,23 ,Chennai
007,Komal Nayak,24,trivendram
008,Bharathi Nambiayar,24,Chennai
We have loaded this file into Pig with the relation name student_details as shown below.

grunt> student_details = LOAD 'hdfs://localhost:9000/pig_data/student_details.txt' USING PigStorage(',')

 as (id:int, name:chararray, age:int, city:chararray);

Tokenizing a String
We can use the TOKENIZE() function to split a string. As an example let us split the name using this function as shown below.

grunt> student_name_tokenize = foreach student_details Generate TOKENIZE(name);

Verification

Verify the relation student_name_tokenize using the DUMP operator as shown below.

grunt> Dump student_name_tokenize;

Output

It will produce the following output, displaying the contents of the relation student_name_tokenize as follows.
({(Rajaiv),(Reddy)})
({(siddarth),(Battacharya)})
({(Rajesh),(Khanna)})
({(Preethi),(Agarwal)})
({(Trupthi),(Mohanthy)})
({(Archana),(Mishra)})
({(Komal),(Nayak)})
({(Bharathi),(Nambiayar)})

Other Delimeters
In the same way, including space [], the TOKENIZE() function accepts double quote [" "], coma [ , ], parenthesis [ () ], star [ * ] as
delimeters.

Example

Suppose there is a file named details.txt with students details like id, name, age, and city. Under the name column this file
contains the first name and the last name of the students separated by various delimeters as shown below.

details.txt
001,"siddarth""Battacharya",22,Kolkata
002,Rajesh*Khanna,22,Delhi
003,(Preethi)(Agarwal),21,Pune
We have loaded this file into Pig with the relation name details as shown below.

grunt> details = LOAD 'hdfs://localhost:9000/pig_data/details.txt' USING PigStorage(',')

as (id:int, name:chararray, age:int, city:chararray);

Now, try to separate the first name and the last name of the students using TOKENIZE() as follows.

grunt> tokenize_data = foreach details Generate TOKENIZE(name);

On verifying the tokenize_data relation using dump operator you will get the following result.
grunt> Dump tokenize_data;
({(siddarth),(Battacharya)})
({(Rajesh),(Khanna)})
({(Preethi),(Agarwal)})