Introduction To Big Data and NoSQL

Introduction to Big Data and NoSQL
SQL Azure Saturday April, 21, 2012

Don Demsak Advisory Solutions Architect EMC Consulting www.donxml.com
Meet Don
Advisory Solutions Architect EMC Consulting
Application Architecture, Development & Design
DonXml.com, Twitter: donxml Email don@donxml.com SlideShare - http://www.slideshare.net/dondemsak
The era of Big Data
How did we get here?

Expensive
Processors Disk space Memory Operating Systems Software Programmers
Monoculture
Limit CPU cycles Limit disk space Limit memory Limited OS Development Limited Software Programmers
Mono-lingual Mono-persistence
Typical RDBMS Implementations

Fixed table schemas Small but frequent reads/writes Large batch transactions Focus on ACID
Atomicity Consistency Isolation Durability
How we scale RDBMS implementations
1st Step Build a relational database
Database
2nd Step Table Partitioning

p1 p2 p3
Database
3rd Step Database Partitioning

Browser Customer #1 Web Tier B/L Tier Database
Browser Customer #2
Web Tier
B/L Tier
Database
Browser Customer #3
Web Tier
B/L Tier
Database
4th Step Move to the cloud?

Browser Customer #1 Web Tier B/L Tier
SQL Azure Federation
Browser Customer #2
Web Tier
B/L Tier
Browser Customer #3
Web Tier
B/L Tier
10
There has to be other ways
11
Polyglot Persistence
12
Polyglot Programmer
13
14
Where Did NoSQL Originate?

1998 - Carlo Strozzi
NoSQL project - lightweight open-source relational DB with no SQL interface
2009 - Eric Evans & Johan Oskarsson of Last.fm wanted to organize an event to discuss opensource distributed databases
15
NoSQL (loose) Definition

(often) Open source Non-relational Distributed (often) dont guarantee ACID
16
Atlanta 2009
No:sql(east) conference
select fun, profit from real_world where relational=false
Billed as conference of no-rel datastores
17
Types Of NoSQL Data Stores
18
5 Groups of Data Models

Relational
Document
Key Value
Graph
Column Family
19
Document Store
Apache Jackrabbit CouchDB MongoDB SimpleDB
XML Databases
MarkLogic Server eXist.
20
Document?
Okay think of a web page...
Relational model requires column/tag Lots of empty columns Wasted space
Document model just stores the pages as is

Saves on space Very flexible.
21
Graph Storage
AllegroGraph Core Data Neo4j DEX
FlockDB
Microsoft Trinity (research project)
http://research.microsoft.com/en-us/projects/trinity/
22
Whats a graph?
Graph consists of
Node (stations of the graph) Edges (lines between them)
FlockDB
Created by the Twitter folks Nodes = Users Edges = Nature of relationship between nodes.
23
Key/Value Stores
On disk Cache in Ram Eventually Consistent
Weak Definition
If no updates occur for a period, eventually all updates will propagate through the system and all replicas will be consistent
Strong Definition
for a given update and a given replica eventually either the update reaches the replica or the replica retires
Ordered
Distributed Hash Table allows lexicographical processing
24
Key/Value Examples
Azure AppFabric Cache Memcache-d VMWare vFabric GemFire
25
Object Databases
Db4o GemStone/S InterSystems Cach Objectivity/DB
ZODB
26
Tabular
BigTable Mnesia Hbase Hypertable
Azure Table Storage

SQL Server 2012
27
Azure Table Storage Demo
28
Big Data
29
Big Data Definition

Volumes & volumes of data Unstructured Semi-structured Not suited for Relational Databases
Often utilizes MapReduce frameworks
30
Big Data Examples

Cassandra Hadoop Greenplum Azure Storage
EMC Atmos
Amazon S3 SQL Azure (with Federations support)
31
Real World Example

Twitter
The challenges
Needs to store many graphs
Who you are following Whos following you Who you receive phone notifications from etc
To deliver a tweet requires rapid paging of followers Heavy write load as followers are added and removed Set arithmetic for @mentions (intersection of users).
32
What did they try?

Started with Relational Databases
Tried Key-Value storage of denormalized lists

Did it work?
Nope
Either good at
Handling the write load Or paging large amounts of data But not both
33
What did they need?

Simplest possible thing that would work Allow for horizontal partitioning Allow write operations to Arrive out of order
Or be processed more than once Failures should result in redundant work
Not lost work!
34
The Result was FlockDB

Stores graph data Not optimized for graph traversal operations Optimized for large adjacency lists
List of all edges in a graph
Key is the edge value a set of the node end points
Optimized for fast read and write

Optimized for page-able set arithmetic.
35
How Does it Work?

Stores graphs as sets of edges between nodes Data is partitioned by node
All queries can be answered by a single partition
Write operations are idempotent

Can be applied multiple times without changing the result
And commutative
Changing the order of operands doesnt change the result.
36
Working With Big Data
37
ACID
Atomicity
All or Nothing
Consistency
Valid according to all defined rules
Isolation
No transaction should be able to interfere with another transaction
Durability
Once a transaction has been committed, it will remain so, even in the event of power loss, crashes, or errors
38
BASE
Basically Available
High availability but not always consistent
Soft state
Background cleanup mechanism
Eventual consistency
Given a sufficiently long period of time over which no changes are sent, all updates can be expected to propagate eventually through the system and all the replicas will be consistent.
39
Traditional (relational) Approach
Extract
Transactional Data Store
Transform
Load
Data Warehouse
40
Big Data Approach

MapReduce Pattern/Framework
an Input Reader Map Function To transform to a common shape (format) a partition function a compare function Reduce Function an Output Writer
41
MongoDB Example
> // map function > m = function(){ ... this.tags.forEach( ... function(z){ ... emit( z , { count : 1 } ); ... } ... ); ...}; > // reduce function > r = function( key , values ){ ... var total = 0; ... for ( var i=0; i<values.length; i++ ) ... total += values[i].count; ... return { count : total }; ...};
> // execute > res = db.things.mapReduce(m, r, { out : "myoutput" } );
42
MongoDB Demo
43
Big Data on Azure

Azure Table Storage
Azure Service Bus
SQL Azure Federations

MongoDB on Azure
http://www.mongodb.org/display/DOCS/MongoDB+on+Azure
Hadoop on Azure
https://www.hadooponazure.com/
44
Using Azure for Computing
Data
Worker Master Job/Task Scheduler Worker Worker
Data
Client
Data
Data
45
Moving to Event Based Architecture

Web Role Web Role Worker Role Worker Role
Web Role
Worker Role
Req
Req
Req
Queue
Web Role
Worker Role
Web Role
Web Role
Monitor queue length against users expectations
Worker Role
Worker Role
46
Aggregate Stores
47
Visualizing Aggregates
ID: 1001 Customer: Ann Line Items
Orders
Customers
32411234 707423234 125145
2 1 1
$48 $56 $24
$96 456 $24
Payment Details
Order Lines
Card: AmEx CC#: 12343 Expiration: 07/2015
Credit Cards
48
Visualizing Aggregates
ID: 1001 Customer: Ann
Line Items
32411234 707423234 125145
2 1 1
$48 $56 $24
$96 456 $24
Payment Details
{ SalesOrdersView:{ ID: 1001, Customer: Ann, LineItems: [] .. . .. } }
Card: AmEx CC#: 12343 Expiration: 07/2015
49
MongoDB on Azure Demo
50
Next Steps
Learn a NoSQL product
Great place to start AppFabric Cache, Azure Table Storage, MongoDB
Pick a new programming language to learn

Not Java or C#/VB Node.js, JavaScript, F#
51
THANK YOU
52

Introduction To Big Data and NoSQL

Uploaded by

Copyright:

Available Formats

Introduction To Big Data and NoSQL

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Introduction To Big Data and NoSQL

Uploaded by

Copyright:

Available Formats

Introduction to Big Data and NoSQL

SQL Azure Saturday April, 21, 2012

DonXml.com, Twitter: donxml Email don@donxml.com SlideShare - http://www.slideshare.net/dondemsak

The era of Big Data

How did we get here?

Typical RDBMS Implementations

How we scale RDBMS implementations

1st Step Build a relational database

2nd Step Table Partitioning

3rd Step Database Partitioning

4th Step Move to the cloud?

SQL Azure Federation

SQL Azure Federation

There has to be other ways

Where Did NoSQL Originate?

NoSQL (loose) Definition

Billed as conference of no-rel datastores

Types Of NoSQL Data Stores

5 Groups of Data Models

Document model just stores the pages as is

Azure Table Storage

Azure Table Storage Demo

Big Data Definition

Often utilizes MapReduce frameworks

Big Data Examples

Real World Example

What did they try?

Tried Key-Value storage of denormalized lists

What did they need?

Not lost work!

The Result was FlockDB

Optimized for fast read and write

How Does it Work?

Write operations are idempotent

Working With Big Data

Traditional (relational) Approach

Transactional Data Store

Big Data Approach

> // execute > res = db.things.mapReduce(m, r, { out : "myoutput" } );

Big Data on Azure

SQL Azure Federations

Using Azure for Computing

Worker Master Job/Task Scheduler Worker Worker

Moving to Event Based Architecture

Monitor queue length against users expectations

32411234 707423234 125145

$48 $56 $24

$96 456 $24

Card: AmEx CC#: 12343 Expiration: 07/2015

32411234 707423234 125145

$48 $56 $24

$96 456 $24

{ SalesOrdersView:{ ID: 1001, Customer: Ann, LineItems: [] .. . .. } }

Card: AmEx CC#: 12343 Expiration: 07/2015

MongoDB on Azure Demo

Pick a new programming language to learn

You might also like