ITBSPEC2: Fundamentals of Enterprise Data Management

Introduction to Data Warehouse

Module 3

Information Technology
Department
Understanding a Data Warehouse
• A data warehouse is a database, which is kept separate from the
organization's operational database.
• There is no frequent updating done in a data warehouse.
• It possesses consolidated historical data, which helps the
organization to analyze its business.
• A data warehouse helps executives to organize, understand, and use
their data to take strategic decisions.
• Data warehouse systems help in:
– the integration of diversity of application systems.
– consolidated historical data analysis.

Information Technology Department

Why a Data Warehouse is Separated from
Operational Databases
• A data warehouses is kept separate from operational databases due
to the following reasons:
– An operational database is constructed for well-known tasks and workloads
such as searching particular records, indexing, etc. In contract, data
warehouse queries are often complex and they present a general form of
data.
– Operational databases support concurrent processing of multiple
transactions. Concurrency control and recovery mechanisms are required for
operational databases to ensure robustness and consistency of the database.
– An operational database query allows to read and modify operations, while
an OLAP query needs only read only access of stored data.
– An operational database maintains current data. On the other hand, a data
warehouse maintains historical data.

Information Technology Department

Data Warehouse Features
• The key features of a data warehouse are discussed below:
– Subject-oriented
– Integrated
– Time Variant
– Non-Volatile

Information Technology Department

Data Warehouse Features
• Subject-Oriented
– A data warehouse is subject oriented because it provides information around
a subject rather than the organization's ongoing operations. These subjects
can be product, customers, suppliers, sales, revenue, etc. A data warehouse
does not focus on the ongoing operations, rather it focuses on modeling and
analysis of data for decision making
• Integrated
– A data warehouse is constructed by integrating data from heterogeneous
sources such as relational databases, flat files, etc. This integration enhances
the effective analysis of data.
• Time Variant
– The data collected in a data warehouse is identified with a particular time
period. The data in a data warehouse provides information from the historical
point of view.

Information Technology Department

Data Warehouse Features
• Non-Volatile
– Non-volatile means the previous data is not erased when new data is added
to it. A data warehouse is kept separate from the operational database and
therefore frequent changes in operational database is not reflected in the
data warehouse

Information Technology Department

Data Warehouse Applications
• Data warehouses are widely used in the following fields:
– Financial services
– Banking services
– Consumer goods
– Retail sectors
– Controlled manufacturing

Information Technology Department

Types of Data Warehouse
• Information processing, analytical processing, and data mining are
the three types of data warehouse applications that are discussed
below:
– Information Processing - A data warehouse allows to process the data stored
in it. The data can be processed by means of querying, basic statistical
analysis, reporting using crosstabs, tables, charts, or graphs.
– Analytical Processing - A data warehouse supports analytical processing of
the information stored in it. The data can be analyzed by means of basic OLAP
operations, including slice-and-dice, drill down, drill up, and pivoting.
– Data Mining - Data mining supports knowledge discovery by finding hidden
patterns and associations, constructing analytical models, performing
classification and prediction. These mining results can be presented using the
visualization tools.

Information Technology Department

Data Warehouse (OLAP) Operational Database (OLTP)
• It involves historical processing • It involves day-to-day
of information processing.
• OLAP systems are used by • OLTP systems are used by
knowledge workers such as clerks, DBAs, or database
executives, managers, and professionals
analysts. • It is used to run the business.
• It is used to analyze the • It focuses on Data in.
business. • It is based on Entity
• It focuses on Information out. Relationship Model.
• It is based on Star Schema, • It is application oriented.
Snowflake Schema, and Fact
Constellation Schema.
• It focuses on Information out.
Information Technology Department
Data Warehouse (OLAP) Operational Database (OLTP)
• It contains historical data. • It contains current data.
• It provides summarized and • It provides primitive and highly
consolidated data. detailed data.
• It provides summarized and • It provides detailed and flat
multidimensional view of data. relational view of data.
• The number of users is in • The number of users is in
hundreds. thousands.
• The number of records • The number of records
accessed is in millions. accessed is in tens.
• The database size is from • The database size is from 100
100GB to 100 TB. MB to 100 GB.
• These are highly flexible. • It provides high performance.

Information Technology Department

 Data Warehousing – Concepts
_____________________________

Information Technology
Department
Data Warehousing - Concepts
• What is Data Warehousing?
– Data warehousing is the process of constructing and using a data
warehouse. A data warehouse is constructed by integrating data
from multiple heterogeneous sources that support analytical
reporting, structured and/or ad hoc queries, and decision
making. Data warehousing involves data cleaning, data
integration, and data consolidations.

Information Technology Department

Data Warehousing - Concepts
• Using Data Warehouse Information
– The information gathered in a warehouse can be used in any of
the following domains:
• Tuning Production Strategies - The product strategies can be well tuned
by repositioning the products and managing the product portfolios by
comparing the sales quarterly or yearly.
• Customer Analysis - Customer analysis is done by analyzing the
customer's buying preferences, buying time, budget cycles, etc.
• Operations Analysis - Data warehousing also helps in customer
relationship management, and making environmental corrections. The
information also allows us to analyze business operations.

Information Technology Department

Data Warehousing - Concepts
• Integrating Heterogeneous Databases
– Query-driven Approach
– Update-driven Approach

Information Technology Department

Query-driven Approach
• It is the traditional approach to integrate heterogeneous
databases. This approach was used to build wrappers and
integrators on top of multiple heterogeneous databases.
These integrators are also known as mediators.

Information Technology Department

Query-driven Approach
• Process of Query-Driven Approach
– When a query is issued to a client side, a metadata dictionary translates the
query into an appropriate from for individual heterogeneous sites involved.
– Now these queries are mapped and sent to the local query processor.
– The results from heterogeneous sites are integrated into a global answer set.
• Disadvantages
– Query-driven approach needs complex integration and filtering processes.
– This approach is very inefficient.
– It is very expensive for frequent queries.
– This approach is also very expensive for queries that require aggregations.

Information Technology Department

Update-driven Approach
• This is an alternative to the traditional approach.
• In update-driven approach, the information from multiple
heterogeneous sources are integrated in advance and are stored in a
warehouse. This information is available for direct querying and
analysis.
• Advantages:
– This approach has the following advantages:
– This approach provide high performance.
– The data is copied, processed, integrated, annotated, summarized and
restructured in semantic data store in advance.
– Query processing does not require an interface to process data at local
sources.

Information Technology Department

Functions of Data Warehouse Tools and Utilities

• The following are the functions of data warehouse tools and

utilities:
– Data Extraction - Involves gathering data from multiple heterogeneous
sources.
– Data Cleaning - Involves finding and correcting the errors in data.
– Data Transformation - Involves converting the data from legacy format to
warehouse format.
– Data Loading - Involves sorting, summarizing, consolidating, checking
integrity, and building indices and partitions.
– Refreshing - Involves updating from data sources to warehouse.

Information Technology Department

Data Warehousing – Terminologies
_____________________________

Information Technology
Department
Metadata
• Metadata is simply defined as data about data. The data that are
used to represent other data is known as metadata.
• For example, the index of a book serves as a metadata for the
contents in the book. In other words, we can say that metadata is
the summarized data that leads us to the detailed data.
• In terms of data warehouse, we can define metadata as following:
– Metadata is a road-map to data warehouse.
– Metadata in data warehouse defines the warehouse objects.
– Metadata acts as a directory. This directory helps the decision support system
to locate the contents of a data warehouse.

Information Technology Department

Metadata Repository
• Metadata repository is an integral part of a data warehouse system.
It contains the following metadata:
– Business metadata - It contains the data ownership information, business
definition, and changing policies.
– Operational metadata - It includes currency of data and data lineage.
Currency of data refers to the data being active, archived, or purged. Lineage
of data means history of data migrated and transformation applied on it.
– Data for mapping from operational environment to data warehouse - It
metadata includes source databases and their contents, data extraction, data
partition, cleaning, transformation rules, data refresh and purging rules.
– The algorithms for summarization - It includes dimension algorithms, data
on granularity, aggregation, summarizing, etc.

Information Technology Department

Data Cube
• A data cube helps us represent data in multiple dimensions. It is
defined by dimensions and facts.
• The dimensions are the entities with respect to which an enterprise
preserves the records.

Information Technology Department

Illustration of Data Cube
• Suppose a company wants to keep track of sales records with the
help of sales data warehouse with respect to time, item, branch,
and location. These dimensions allow to keep track of monthly sales
and at which branch the items were sold. There is a table associated
with each dimension. This table is known as dimension table. For
example, "item" dimension table may have attributes such as
item_name, item_type, and item_brand.

Information Technology Department

Illustration of Data Cube
• The following table represents the 2-D view of Sales Data for a
company with respect to time, item, and location dimensions.

Information Technology Department

Illustration of Data Cube
• The sales for New Delhi are shown with respect to time, and item dimensions
according to type of items sold. If we want to view the sales data with one more
dimension, say, the location dimension, then the 3-D view would be useful. The 3-
D view of the sales data with respect to time, item, and location is shown in the
table below:

Information Technology Department

Illustration of Data Cube
• The above 3-D table can be represented as 3-D data cube as shown
in the following figure:

Information Technology Department

 Multidimensional
Database Model
Customer Store
Store

Time Time

SALES FINANCE

Product

The data is found at the intersection of dimensions.

Data Mart
• Data marts contain a subset of organization-wide data that is
valuable to specific groups of people in an organization. In other
words, a data mart contains only those data that is specific to a
particular group.
• For example, the marketing data mart may contain only data related
to items, customers, and sales. Data marts are confined to subjects.

Information Technology Department

Points to Remember About Data Marts

• Windows-based or Unix/Linux-based servers are used to implement

data marts. They are implemented on low-cost servers.
• The implementation cycle of a data mart is measured in short
periods of time, i.e., in weeks rather than months or years.
• The life cycle of data marts may be complex in the long run, if their
planning and design are not organization-wide.
• Data marts are small in size.
• Data marts are customized by department.
• The source of a data mart is departmentally structured data
warehouse.
• Data marts are flexible.

Information Technology Department

The following figure shows a graphical representation of data marts

Information Technology
Department
Virtual Warehouse
• The view over an operational data warehouse is known as virtual
warehouse. It is easy to build a virtual warehouse. Building a virtual
warehouse requires excess capacity on operational database servers

Information Technology Department

Data Mining
 Data mining functions
• Associations
– 85 percent of customers who buy a certain brand of wine also buy a certain
type of pasta
• Sequential patterns
– 32 percent of female customers who order a red jacket within six months
buy a gray skirt
• Classifying
– Frequent customers are those with incomes about $50,000 and having two
or more children
• Clustering
– Market segmentation
• Predicting
– predict the revenue value of a new customer based on that personal
demographic variables
End!

Information Technology
Department