Data Science

Introduction
By Sagar Bhilare
M. Sc Artificial Intelligence, BE- Mechanical Engineering
7 Years experience in Data Science mentoring and Product
development
What is Data Science
Data science is the field of study that combines domain expertise, programming
skills, and knowledge of mathematics and statistics to extract meaningful insights
from data.
What is the most important
component of Data Science?
Why do you think organizations need
Data Scientist?
Decision making
Some Decision to make!!
- Should I carry an Umbrella?
- Which color cloth should I wear?
- Which route to choose for office?
- Choosing a gift for someone?
Some More decision to make
How do you?
- Decide whether a patient is ill based on some symptoms
- Decide whether to approve loan to a person based on his borrowing behavior
- Decide a strategy to improvise customer retention
- Decide whether a email is spam or not spam
- Decide and predict if a person may choose to leave a service
- Decide and recommend a best possible product/service or information to a user
Some Examples of Data Driven
Decision making
- OTT Platforms
- E Commerce Websites
- Government Institutions
- Cab and pool services
- Hotel bookings and travel tourism
- Healthcare sector
- Pharmacy sector
- Automotive sector
- Social Media Content
Roles of a Data Science professional
- Database administrator
- Data Scientist
- Machine Learning Engineer
- AI developer
- Tableau Developer
- BI developer
- Deep Learning Engineer
Is there any hype created for Data
Science?
Data Created per day by an
Ecommerce website
Flipkart
- Flipkart sees 1.6 million users per second on Day 1 of festive sale (Economic Times)
- Flipkart gets 10 terabytes of user data each day from browsing, searching, buying
or not buying, as well as behavior and location. This jumps to 50 terabytes on Big
Billion Day sales days.
- What do you think flipkart does with this data?
- Is it Humanly possible to analyze this large chunk of data?
What is Data
Data is a collection of facts, such as numbers, words, measurements, observations or
just descriptions of things.

- What all forms of data have you ve ever encountered?

Database
Database is a collection of Data

Can You give any examples of databases?

Relational Database
RDBMS
A relational database manages data in tables.
Databases are managed by a relational database management system (RDBMS).
An RDBMS supports a database language to create and delete databases and to
manage and search data.
The database language used in almost all DBMSs is SQL.
What is SQL?
SQL stands for Structured Query Language
SQL lets you access and manipulate databases
MYSQL
-MySQL is an open-source relational database management system (RDBMS)
- It is used to work upon databases and tables inside the databases
MY SQL Database
- It is a collection of tables
My SQL Table
Table is a systematic arrangement of data in a MYSQL database in the form of rows
and columns
Each row is a observation
Each column is a attribute
Installation
Visit: https://www.mysql.com/downloads/
- Click MySQL Community Downloads
- click MySQL Installer
- choose the version and download
Commonly used Datatypes in MySQL
- int()
- float()
- bool()
- varchar()
- char()
- Date()
- DateTime()
Types of SQL
Here are five types of widely used SQL queries.
Data Definition Language (DDL)
Data Manipulation Language (DML)
Data Control Language(DCL)
Transaction Control Language(TCL)
Data Query Language (DQL)
What is DDL?
Data Definition Language helps you to define the database structure or schema
- CREATE: used to create database, table, procedure views etc
- ALTER: Alters command allows you to alter the structure of the database.
- DROP: Drop commands remove tables and databases from RDBMS.
- TRUNCATE: This command used to delete all the rows from the table and free the
space containing the table.
CREATE
create database school
CREATE TABLE Student ( Rollno int, FirstName varchar(255), LastName
varchar(255), Class int, Hometown varchar(255));
ALTER
- ALTER TABLE Student
ADD marks int();

- ALTER TABLE table_name

DROP COLUMN column_name;

- ALTER TABLE table_name

MODIFY COLUMN column_name datatype;
DROP
DROP TABLE table_name;
DROP database database_name;
TRUNCATE
TRUNCATE TABLE table_name;
constraints
Constraints are rules associated with data
Constraints can be specified when the table is created with the CREATE TABLE statement, or
after the table is created with the ALTER TABLE statement.

Types of Constraints
- NOT NULL - Ensures that a column cannot have a NULL value
- UNIQUE - Ensures that all values in a column are different
- PRIMARY KEY - A combination of a NOT NULL and UNIQUE. Uniquely identifies each row in a table
- FOREIGN KEY - Prevents actions that would destroy links between tables
- CHECK - Ensures that the values in a column satisfies a specific condition
- DEFAULT - Sets a default value for a column if no value is specified
NOT NULL
CREATE TABLE teachers (
ID int NOT NULL,
LastName varchar(255) NOT NULL,
FirstName varchar(255) NOT NULL,
Age int
);

ALTER TABLE teachers

MODIFY Age int NOT NULL;
UNIQUE Constraint
CREATE TABLE teachers(
ID int NOT NULL,
LastName varchar(255) NOT NULL,
FirstName varchar(255),
Age int,
UNIQUE (ID)
);

ALTER TABLE teachers

ADD CONSTRAINT UC_teacher UNIQUE (ID);
PRIMARY KEY
CREATE TABLE teachers (
ID int NOT NULL,
LastName varchar(255) NOT NULL,
FirstName varchar(255),
Age int,
PRIMARY KEY (ID)
);

ALTER TABLE teachers

ADD PRIMARY KEY (ID);
Foreign Key
CREATE TABLE room(
room_no int NOT NULL,
student_id int
teacher_id int,
PRIMARY KEY (room_no),
FOREIGN KEY (teacher_id) REFERENCES teachers(id)
);

ALTER TABLE room

ADD FOREIGN KEY (student_id) REFERENCES student(Rollno);
CHECK
CREATE TABLE Persons (
ID int NOT NULL,
LastName varchar(255) NOT NULL,
FirstName varchar(255),
Age int,
CHECK (Age>=18)
);

ALTER TABLE Persons

ADD CHECK (Age>=18);
DEFAULT
CREATE TABLE Persons (
ID int NOT NULL,
LastName varchar(255) NOT NULL,
FirstName varchar(255),
Age int,
City varchar(255) DEFAULT 'Sandnes’
);

ALTER TABLE Student

ALTER FirstName SET DEFAULT ’Unknown';
Task 1
1. create a new database “Neopro” and activate it to store database object such as
tables.
2. create 3 different tables - sales, marketing and accounts and each table should
have employee_id, first name, last name, gender and age.
3. add proficiencylevel column to all the tables using alter command and set
proficiencylevel column default =1
4. In each table we have to alter employee_id with primarykey constraint
Task 1 Contd….
5. We have to alter the proficiencylevel column in each table with check constraint and the
proficiencylevel of employees should be between 1 and 5
6. We have to modify the age column in each table with check constraint and the age of
employees should be between 21 and 55
7. Add a new table called as project in that table add the columns, project id primary key,
project name, project duration, project client name.
8. add a column project id in each of the first three tables on sales, marketing and accounts.
9. each of the project id from these three tables will refer to project id of project table
10. add a new column work location with default location being unknown in the project
table
Jatin University

School_id location_id Studentid

school_name locaton_name student_name
School_ranking country_id school_id
location_id bus_id
location_id
School_details locations student_information

Bus_no driver_id college_id country id

Bus_type driver_name college_name country_name
Driver_id location_id college_ranking
Bus_fees bus_id placement_percent
student_id
country_id
Bus information Driver information college_info
What is DML?
- Data Manipulation Language (DML) allows you to modify the database instance by
inserting, modifying, and deleting its data. It is responsible for performing all types
of data modification in a database.
- Here are some important DML commands in SQL:
INSERT: It is used to insert new information in a table
UPDATE: It is used to update a particular information a table
DELETE: It is used to delete a particular information in a table
INSERT
INSERT INTO students ( Rollno, FirstName, LastName, Class, Hometown);
VALUES (1 ’Ajay', ’Kumar’, 5 ’Mumbai’),
(2 ’Amit’, ’Mane’, 7 ’Delhi’),
(3 ’Aditi’, ’Sharma’, 8 ’Bangalore’);
Task 1 Contd….
7. Insert 10 values into Sales Tables the sales employee id begins this NEOS101
8. Insert 10 values into marketing Tables the sales employee id begins this
NEOM101
9. Insert 10 values into account Tables the sales employee id begins this NEOA101
UPDATE
UPDATE students
SET Class = 10
WHERE Hometown = ’Delhi’;
DELETE
DELETE FROM Student WHERE Hometown = ‘Bangalore';
Task 1 Contd…..
10. update the proficiency level to 3 for the employees with the following employee
codes:
NEOS104, NEOS105, NEOM107, NEOM110, NEOA102, NEOA101
11. delete the record NEOS108, NEOS104, NEOM103, NEOM105, NEOA101,
NEOA108
What is DQL?
- Data Query Language (DQL) allows you to perform queries on the data within
schema objects.-

Here are some important DQL commands in SQL:

Select: It is used to retrieve the data from a database table
SELECT
Select * from students
SELECT Rollno, firstname, lastname FROM students;
Some more important commands
Distinct: it is used to show unique values
Where: it is used to fetch a data with respect a particular condition
LIKE: It is used to check a specific pattern in String values
AND /or/ NOT: It is used to apply multiple conditions
In: IN operator allows you to specify multiple values in a WHERE clause.
Between: It is used to specify a range of values to check for a condition I where
clause
Distinct
SELECT DISTINCT class FROM student;

Select count(distinct class) from student

WHERE
Select * from teachers where id>103;

A where statement can be combined si followed up by some conditional operators

like
AND
OR
IN
BETWEEN
LIKE
Select * from student where name like ‘a%’
Select * from student where name like ‘%a’
Select * from student where name like ‘_a%’
AND/ OR / NOT
Select * from student where name like ‘a%’ and class<8

Select * from student where name like ‘a%’ or name like ‘v%’

Select * from student where name not like ‘a%’ or name like ‘v%’
IN
Select * from student where hometown in (‘Mumbai’, ‘Bangalore’, ‘Pune’)
Select * from student where hometown not in (‘Mumbai’, ‘Bangalore’, ‘Pune’)
BETWEEN
Select * from student Where class between 7 and 10
Sorting the data
We can sort the data in mysql with the help of command order by and keyword
desc

Select * from student order by Class

Select * from student order by class desc
limit
IT is used to show limited number of items in a MYSQL table

Select * from Student limit 3

Aggregations
Avg: It is used to calculate average or mean
Sum: It is used to calculate the sum of values
Count: It is used to count the no of rows
Min: It finds the minimum value
Max: It finds the maximum value
avg
Select avg(class) from student;
sum
Select sum(class) from student;
count
Select count(RollNo) from student;
Min and max
Select min(class) from student

Select max(class) from student

Task 1 Contd….
13. In NeoPro Database identify the sales employees whose proficiency level is greater than
4
14. Identify all females from marketing department whose name starts with p
15. Identify all females from accounts department whose name ends with a and proficiency
level between 2 and 4
16. Find the count of females in Accounts department
17. Organization is not able to achieve the desired targets hence the management has
decided to hire more sales employee in the team but at the cost of old employees with a
proficiency score less than 3. Update the records in sales table remove the old values and
add the same no of values in the sales table and assign proficiency score more than or equal
to 4.
Creating back ups
Create table student_bkp select * from student
Create table teacher_bkp like teachers
Merging the tables
Insert into student select * from student_bkp
Task 1 contd…
18. Company NEO Pro has decided to create a backup of all the data help the
organization achieve this task
19. Delete the records from NEOA101 to NEOA105
20. As most of people from accounts department has left the organization so
company has decided to store the accounts data along with marketing data
21. Delete the accounts data after creating a new table with the same structure as
accounts
Joins
Joins are used to combine two tables on a common column(key).
Types of Joins
1. Inner Join
2. Outer Join
- left join
- Right join
- Full outer join
3. Cross Join
4. SELF Join
Inner Join
SELECT column_name(s)
FROM table1
INNER JOIN table2
ON table1.column_name = table2.column_name;
Right Join
SELECT column_name(s)
FROM table1
RIGHT JOIN table2
ON table1.column_name = table2.column_name;
Left join
SELECT column_name(s)
FROM table1
LEFT JOIN table2
ON table1.column_name = table2.column_name;
UNION
It is used to combine the result set of two or more select statements.
Every select statement within union must have same number or columns and the
columns must be of similar data types columns in the select statements must be in
same order.
SELECT column_name(s) FROM table1
UNION
SELECT column_name(s) FROM table2;
Intersection
CREATE TABLE A(id INT);
CREATE TABLE B(id INT);
INSERT INTO A VALUES(1),(2),(3),(4);
INSERT INTO B VALUES(3),(4),(5),(6);
SELECT id FROM A INNER JOIN B USING(id);
Views
View is a virtual table based on the result-set of an SQL statement.
We can show a part of data and it can be seen again and again we have to just call
the view to see the data.

create view it_emp as select first_name, last_name, department_name from

employees join departments using(department_id) where department_name= "IT";
Indexing
Indexes are used to retrieve data from the database more quickly than otherwise.
The users cannot see the indexes, they are just used to speed up searches/queries.

Create index myindex on employees(employee_id)

FULL Outer JOIN
It is a combination of Right and Left Join on a common
column

SELECT * FROM
table1 LEFT JOIN table2 ON table1.id = table2.id
UNION
SELECT * FROM
table1 RIGHT JOIN table2 ON table1.id = table2.id
CROSS JOIN
A CROSS JOIN produces a cartesian product between the two
tables,
returning all possible combinations of all rows.
It has no ON clause because you're just joining everything to
everything

SELECT column_name(s)
FROM table1
CROSS JOIN table2;

.
Self Join
SELECT s1.col_name, s2.col_name...
FROM table1 s1, table1 s2
WHERE s1.common_col_name = s2.common_col_name;
Group By
It is used to summarize the data based on some category.
Summarization can happen with aggregation functions such as mean, count, sum
min and max

SELECT column_name(s)
FROM table_name
WHERE condition
GROUP BY column_name(s)
ORDER BY column_name(s);
TASK 2
Fetch the names and ids of the employees who are working in the IT department.
select employee_id, first_name, department_name from employees join
departments on employees.department_id= departments.department_id;
Fetch the first name, last name, job id, job title, minimum salary and maximum
salary of all employees
select first_name, last_name, department_name, min_salary, max_salary from
employees join departments on employees.department_id=
departments.department_idjoin jobs on employees.job_id= jobs.job_id ;
Identify the top 10 cities which have the largest number of employees
select city, count(employee_id) from employees join departments
using(department_id) join locations using(location_id) group by city order by
count(employee_id) desc;
Fetch the employee ids and names of all employees whose last working day in the
organization was 1999-12-31
select employee_id, first_name from employees join job_history
using(employee_id) where end_date= "1999-12-31";
Fetch the employee id , first name, department name and total experience of all
employees who have completed at least 25 years in the organization
select first_name, salary, year(now())- year(hire_date) as Total_Experience from
employees where year(now())- year(hire_date)>25 ;
Fetch the details of top 3 countries where most of the employees are working
select country_name, count(*) from employees join departments
using(department_id) join locations using(location_id) join countries
using(country_id)group by country_name order by count(*) desc limit 3;
Fetch the details of those employees who have completed 10 years in the
organization
select first_name, salary, year(now())- year(hire_date) as Total_Experience from
employees where year(now())- year(hire_date)>10 ;
Fetch the department wise cost to the company
select department_name, sum(salary) as TotalCost from employees join
departments using(department_id) group by department_name order by
sum(salary) desc ;
Fetch the details of employees whose salary is greater than average salary
select first_name, last_name , salary from employees where salary> (select
avg(salary) from employees);
Find the names of all employees whose salary is greater than 60% of their
departments total salary bill.
select first_name, last_name , salary from employees where salary in (select
sum(salary) from employees group by department_id having
sum(salary)>0.6*sum(salary) ) order by department_id ;
 Subqueries
A MySQL subquery is a query nested within another query
such as SELECT, INSERT, UPDATE or DELETE.
It is nested inside the where or having clauses.
Database Engine executes the inner query and
uses the result to calculate the result of the outer query.

Select language from CountryLanguage where

CountryCode= (select code from country where Name= ‘Finland’);
Scalar Subqueries
Scalar subqueries act as simple, singular value exxpressions

Select Country.name, 100*Country.Population/(select sum(Population) from

Country)
As PercentWorldPopulation from Country;
Aggregations using scalar subqueries
Select name, (select count(*) from city where CountryCode = Code) as Cities,
(Select count(*) from CountryLanguage where CountryCode= Code) as Languages
From Country;
SQL Arithmetic Operators
Operator Use
+ ADD
- Subtract
* Multiply
/ Divide
% Modulo
SQL BitWise Operators
Operator Use
& Bitwise And
| Bitwise OR
^ Bitwise Exclusive or
SQL Comparison Operators
Operator Use
= Equal to
> Greater than
< Less than
>= Greater than or equal to
<= Less than or equal to
<> Not equal to
SQL Logical Operators
Operator Use
And, or, not Comparing two conditions
In, between, exists Checking if values lie in a range
of values
like Checks the pattern
all True if all values meet the
condition
any True if any value meet the
condition
Aggregations in the from clause
Select avg(cont_sum) from( select Continent, sum(Population) as cont_sum from
Country group by Continent) as t;

Select count(employee_id) from employees group by department_id

having
Having clause is similar to where but as we can’t use where clause with aggregate
functions hence we have having

Select deparment_name, count(employee_id) from employees join departments on

employees.department_id= departments.department_id group by
department_name Having count(employee_id) >1;
Exists
It is used to check if any record exists in a subquery
Select employee_id from employees where exists(select employee_id from
job_history where job_id= 'ST_CLERK' );
any
It compares the single column value with range of other value such that if any of
subqueries meet the condition.

select country_name from countries where country_id= any( select country_id from
locations where city in ('Tokyo', 'Venice'));
all
It compares the single column value with range of other value such that if all of
subqueries meet the condition.

SELECT * FROM employees WHERE salary > ALL(SELECT avg(salary)FROM employees

GROUP BY department_id);
Tasks
Find the employee id, first names, last names and department names of all employees in
the organization.
select first_name, last_name, Department_name from employees join epartments
using(department_id);
Find the names and salaries of the employees whose salaries are greater than the average
salary.
select first_name, last_name , salary from employees where salary> (select avg(salary) from
employees);
Find the name and salary of the employees who have a salary higher than the salary of all IT
Programmers(JOB_ID='IT_PROG'), sort the salaries from lowest to highest.
select first_name, last_name, salary from employees where salary> (select max(salary)
from employees where JOB_ID='IT_PROG');
Find the employee with the minimum salary in the employee's table. Arrange the records in
ascending order.
Select first_name, salary from employees where salary = (select min(salary) from
employees)
Windows Functions
● A window function performs a calculation across a set
of table rows that are somehow related to the current
row

● A ‘window’ refers to an individual group or partition

within a result set

● Frequently-used SQL aggregation functions like SUM,

COUNT etc.

● Examples of windows functions

ROW_NUMBER
RANK
DENSE_RANK
LEAD
LAG
etc..
● Uses of OVER() and PARTITION BY clauses to do
advance level grouping https://www.gooddata.com/sites/default/files/sum.png

98
OVER() & PARTITION BY clauses
Example 1:
OVER() SELECT <column list> <AGGR_FUNC> OVER()
FROM <TABLENAME>;
● Indicates that a function will be applied to all the rows
returned by the query

● OVER() can either be blank or it can contain PARTITION Example 2:

BY
SELECT <column list> <AGGR_FUNC> OVER
(PARTITION BY <column>) FROM
<TABLENAME>;
PARTITION BY (<column name>)

● Divides the data (result from SQL query) into a set of

partitions and computation is performed on each
partition

99
OVER() & PARTITION BY clauses
Example 1:
OVER() SELECT <column list> <AGGR_FUNC> OVER()
FROM <TABLENAME>;
● Indicates that a function will be applied to all the rows
returned by the query

● OVER() can either be blank or it can contain PARTITION Example 2:

BY
SELECT <column list> <AGGR_FUNC> OVER
(PARTITION BY <column>) FROM
<TABLENAME>;
PARTITION BY (<column name>)

● Divides the data (result from SQL query) into a set of

partitions and computation is performed on each
partition

10
0
For the windows functions queries, use
the world database and use the table as
“city”
ROW_NUMBER()

● Displays a row number starting from 1 for every country

code on which the data is partitioned

● When the country code changes, the row number for

the new country code starts from 1 again
select ID,Name,CountryCode,District,
Population,

row_number() over(partition by
CountryCode) as rownumber

from city;

10
2
RANK()

● Displays the rank for every country code on which the

data is partitioned

● When partition changes, the next rank is the next

number after the total records of the first partition
select ID,Name,CountryCode,District,
Population,

rank() over(partition by CountryCode order

by Population desc) as ranks

from city;

10
3
DENSE_RANK()

● Displays the rank for every country code on which the

data is partitioned

● When partition changes, the next rank is the next select ID,Name,CountryCode,District,
number after the previous rank Population,

dense_rank() over(partition by CountryCode

order by Population desc) as Dense_ranks

from city;

10
4
LAG()

● For the current record, displays the previous record's

value

select ID,Name,CountryCode,District,
Population,

lag(Population) over(partition by
CountryCode) as lag_Population

from city;

10
5
LEAD()

● For the current record, displays the next record's value

select ID,Name,CountryCode,District,
Population,

lead(Population) over(partition by
CountryCode order by ID ) as
lead_Population

from city;

10
6
MIN(), MAX(), SUM(), AVG()

● Display the minimum, maximum, total and average select ID, Name, CountryCode,Population,
value of a numeric column for every group
max(Population) over(partition by
CountryCode) as maxPopulation,

min(Population) over(partition by
CountryCode) as minPopulation,

sum(Population) over(partition by
CountryCode) as sumPopulation,

avg(Population) over(partition by
CountryCode) as avgPopulation

from city;

10
7
Stored Procedure
MySQL stored procedures are pre-compiled SQL statements stored in a database,
so the code can be reused over and over again.

The stored procedure is SQL statements wrapped within the CREATE PROCEDURE
statement
 A set of pre-compiled SQL statements
Every procedure has a unique name, input and output parameters, and SQL
statements
Can be called multiple times from different applications (front-end, back-end
and/or from other stored procedures / triggers)
Has various uses – like data analysis and impute, add / change / delete / select data
from table(s)
Executed manually by using the procedure name from application with the relevant
parameters
Some databases support recursion
Create Procedure Procedure_Name( )
Begin
SQL Queries..
End

The SQL Queries and code must be written between BEGIN and END keywords.
To create the procedure

Log into MySQL using the credentials

Select a database
Click on the ‘Create new Stored Procedure’ icon
A new window will open
Using the “StoredProcedures.sql”, write the entire content here
Compile the procedure by clicking the “Apply” button
To execute a procedure in MySQL
Log into MySQL using the credentials
Select a database
From the query window, type
call <procedure name> and execute
Benefits of a stored procedure:
Easy to maintain
Secure
Reduce the Network Traffic
Triggers
A set of pre-compiled SQL statements that is automatically fired when an event occurs on a
table
It is a special type of stored procedure that is invoked automatically in response to an
event.
Each trigger is associated with a table, which is activated on any DML statement such
as INSERT, UPDATE, or DELETE.
Triggers are created on a SQL table
The events are
BEFORE / AFTER INSERT
BEFORE / AFTER UPDATE
BEFORE / AFTER DELETE
Triggers can be of 2 types
ROW LEVEL
o Executed each time a row is affected by a I/U/D statement. If 10 rows are
affected, trigger is executed 10 times; one for each row
o If no rows are affected, trigger is not fired

STATEMENT LEVEL
o Executed only once for any I/U/D statement
o If no rows are affected, trigger is not fired
CREATE TRIGGER trigger_name
(AFTER | BEFORE) (INSERT | UPDATE | DELETE)
ON table_name FOR EACH ROW
BEGIN
--variable declarations
--trigger code
END;
To create a trigger in MySQL
We will create an AFTER UPDATE trigger that will create a log entry each time a
record in the table ‘product’ gets updated
For this, we will also create a new log table

Log into MySQL using the credentials

Select a database
Use the ‘Query’ window
Use the file “triggers.sql”, write the entire content here and execute
Update any product record at random
Check the log table
Benefits of Triggers:
Easy to maintain.
Helps us to keep a log of records like maintaining audit trails in tables.
Helps us to validate data even before they are inserted or updated.
Provides an alternative way to check the integrity of data.
Increases the performance of SQL queries because it does not need to compile each time the
query is executed.
Reduces the client-side code that saves time and effort.