Storm Persistence and Real-Time Analytics

Aerospike aer . o . spike [air-oh- spahyk]
noun, 1. tip of a rocket that enhances speed and stability
STORM
PERSISTENCE
AND REAL-TIME
ANALYTICS
APRIL 1, 2014
IN-MEMORY NOSQL DATABASE
brian@aerospike.com

Follow
Join Us!
< meta description=“Aerospike is a
Rule CaptainWord {
strings:
$header = {D0 CF 11 E0 A1 B1 1A E1}
$author = {00 00 00 63 61 70 74 61 69 6E 00}
condition:
$header at 0 and $author
Content=“malware,Exec Code, Overflow, ExecCode Bypass”
Content’=“ Java 0day. Gh0st RAT, Mac Control RAT, MS09-027”
Content=“Hosh Hewer, Jenwediki yighingha iltimas qilish Jediwili”
tag
<head>
<meta name=“”>
<meta desc=“”>
</head>
© 2014 Aerospike. All rights reserved. Pg. 2
“keywords”
<TITLE>
✓ I’M ATTENDING
@Aerospikedb
Real-time analytics with
Storm and Aerospike
Brian Bulkowski
Founder and CTO
Aerospike

Streaming architecture
© 2014 Aerospike. All rights reserved. | Pg. 3
Data
Warehouse,
Hadoop
Cluster
Real-time
Interactions
Server
Batch Analytics
•  User segmentation
•  Location patterns
•  Similar audience
Real-time Interactions
•  Frequency caps
•  Recent ads served
•  Recent search terms
User
Data
Streaming
(Storm)
Hadoop

Why not other databases?
➤  Database requests in bolts
➤  Flash optimized
§  Do you need more than 30G ?
➤  Read / write optimized
➤  Faster & more reliable than
than Kafka (Cassandra based)
➤  Faster than Mongo
➤  More scale than Redis
© 2012 Aerospike. All rights reserved. Confidential | Pg. 4

Examples
➤  Recommendations
§  Multiple recommendation systems
§  Multi-arm bandit
§  https://github.com/tdunning/
storm-counts/wiki/Bayesian-Bandit
➤  Simple fraud counts
§  Store recent requests for payment
§  Store recent users
§  Calculate fraud scores, drop events
if past threshold

Aerospike Bolts
➤  Aerospike has speed, reliability, scale for Storm
§  Free version at http://aerospike.com/
§  Internap – free high performance SSD servers for trial
➤  Bolts available on github
§  https://github.com/aerospike/storm-aerospike
➤  EnrichBolt
§  Add fields from column after looking up a key
➤  PersistBolt
§  Store fields based on a key
➤  Benefits
§  In memory with FLASH
§  Clustered for high performance
§  HA state matches Storm’s stateless model

Follow
Join Us!
Rule CaptainWord {
strings:
$author = {00 00 00 63 61 70 74 61 69 6E 00}
condition:
#HashTags
tag
<head>
<meta name=“”>
<meta desc=“”>
</head>
© 2012 Aerospike. All rights reserved. Confidential Pg. 7
“keywords”
<TITLE>
✓ I’M ATTENDING
@Aerospikedb
The power of Flash

OTHER DATABASE
OS FILE SYSTEM
PAGE CACHE
BLOCK INTERFACE
SSD HDD
BLOCK INTERFACE
SSD SSD
OPEN NVM
SSD
OTHER
DATABASE
AEROSPIKE FLASH OPTIMIZED
IN-MEMORY DATABASE
Ask me and I’ll tell you the answer.Ask me. I’ll look up the answer and then tell it to
you.
AEROSPIKE
HYBRID MEMORY SYSTEM™
Flash-optimization Delivers Disruptive Performance

DRAM & HDD SSD & DRAM
Storage /server 180 GB (196 GB Server) 2.4 TB (4 x 700 GB)
TPS /server 500,000 500,000
Cost /server $8,000 $11,000
Server costs $1,488,000 $154,000
Power /server 0.9 kW 1.1 kW
Power (2 years) $0.12 per kWh ave.
US
$352,000 $32,400
Maintenance (2 years) $3,600 /
server
$670,000 $50,400
Total $2,510,000 $236,800
…at 1/10 the hardware cost
Actual customer analysis
500K TPS
10 TB Storage
2x Replication
186 SERVERS 14 SERVERS
OTHER DATABASES
ONLY

Measure your drives!
Aerospike Certification Tool (ACT)
http://github.com/aerospike/act
Transactional database workload
Reads: 1.5KB
(can’t batch / cache reads, random)
Writes: 128K blocks
(log based layout)
(plus defragmentation)
Turn up the load until
latency is over required SLA

Micron P320h – ACT results
[root@144.bm-general.dev.nym2 act]#
latency_calc/act_latency.py -l
actconfig_micron_75x_1d_rssdb_20130503232823.out
trans device %>(ms) %>(ms)
hour 1 8 64 1 8 64
----- ------ ------ ------ ------ ------
1 0.17 0.00 0.00 0.03 0.00 0.00
2 0.17 0.00 0.00 0.03 0.00 0.00
3 0.18 0.00 0.00 0.03 0.00 0.00
4 0.18 0.00 0.00 0.03 0.00 0.00
5 0.18 0.00 0.00 0.03 0.00 0.00
6 0.19 0.00 0.00 0.04 0.00 0.00
150K read IOPS @ 1.5K
225MB writes @ 128K
225MB reads @ 128K
$8/GB

Test data – the next generation
6K reads per second, 9MB/sec write load
> 1 ms > 8 ms > 64 ms
Intel s3700, 20% OP - 6k iops 1.6 0 0 ($3/GB)
Intel s3700, 20% OP - 12k iops 5.4 0 0
Intel s3700, 20% OP - 24k iops 12.29 0 0
Intel s3700, NO OP - 24k iops 15.33 0 0
FusionIO Iodrive 2 – 6k iops 2.63 0.01 0 ($8/GB)
FusionIO iodrive 2 – 12k iops 7.32 0.1 0

Test data – the previous generation
> 1 ms > 8 ms > 64 ms
Intel X25-M + w/No OP (160G): 17.9% 0.6% 0.4%
Intel X25-M + OP (126G):            3.4% 0.1% 0.08%
OCZ Deneva 2 SLC + OP (95G): 0.9% 0.08% 0%
Samsung SS805 (100G):       2.0% 0.09% 0%
Intel 710 + OP (158G): 4.0% 0.01% 0%
Intel 320 + OP (126G): 5.6% 0% 0%
OCZ Vertex 2 + OP (190G): 6.3% 0.5% 0.01%
SMART XceedIOPS + OP (158G):     5.4% 0.4% 0%
Intel 510 + OP (95G): 6.2% 4.0% 0.03%
Micron P300 + OP (79GB):       1.3% 1.0% 0.7%

Test data – the previous generation
> 1 ms > 8 ms > 64 ms
OCZ Deneva 2 SLC + OP (95G): 3.2% 0.4% 0%
Samsung SS805 (100G): 10.1% 0.8% 0.02%
Intel 320 + OP (126G): 22.0% 0.3% 0.03%
OCZ Deneva 2 MLC (Sync) 8.8% 0.6% 0.06%
OCZ Vertex 2 + OP (190G): 27.6% 4.6% 0.4%
SMART XceedIOPS + OP (158G): 24.5% 5.4% 1.0%

Follow
Join Us!
Rule CaptainWord {
strings:
$author = {00 00 00 63 61 70 74 61 69 6E 00}
condition:
tag
<head>
<meta name=“”>
<meta desc=“”>
</head>
“keywords”
<TITLE>
✓ I’M ATTENDING
@Aerospikedb
Why Aerospike ?

© 2013 Aerospike. All rights reserved. Confidential. 16
➤  Key Value API
➤  Real-time Performance
➤  Read/Write Workloads
➤  Clustering
➤  High Availability
➤  Commodity Hardware
➤  RAM + Flash
➤  XDR
Distributed Key Value Database +
Global Data Management

Challenges
1.  Handle extremely high rates of persistent read/write
transactions
2.  Avoid hot spots to maintain tight latency SLAs
3.  Provide immediate consistency with replication
4.  Allow long running tasks with transactions
5.  Scale linearly as data sizes increase
6.  Add capacity with no service interruption

Aerospike: the gold standard for high throughput,
low latency, high reliability transactions
Performance
• Over ten trillion transactions per
month
• 99% of transactions faster than 2
ms
• 150K TPS per server
Scalability
• Billions of Internet users
• Clustered Software
• Automatic Data Rebalancing
Reliability
• 50 customers; zero service down-
time
• Immediate Consistency
• Rapid Failover; Data Center
Replication
Price/Performance
• Makes impossible projects
affordable
• Flash-optimized
• 1/10 the servers required

10x Performance
0
50,000
100,000
150,000
200,000
250,000
300,000
350,000
Balanced Read-Heavy
Aerospike Cassandra MongoDB Couchbase 2.0*
*We were forced to exclude Couchbase...since when run with either disk
or replica durability on it was unable to complete the test.”
– Thumbtack Technology
0
2.5
5
7.5
10
0 50,000 100,000 150,000 200,000
AverageLatency,ms
Throughput, ops/sec
Balanced Workload Read Latency
Aerospike
Cassandra
MongoDB
0
4
8
12
16
0 50,000 100,000 150,000 200,000
AverageLatency,ms
Throughput, ops/sec
Balanced Workload Update Latency
Aerospike
Cassandra
MongoDB
HIGH THROUGHPUT LOW LATENCY
Throughput,TPS

High Availability
1 32 4 5 Phases
1) 100KTPS – 4 nodes
2) Clients at Max
Aerospike Node Specs:
CentOS 6.3
Intel i5-2400@ 3.1 GHz (Quad core)
16 GB RAM@1333 MHz

➤ Hard to
Maintain
➤ Performance
Better than the Competition
➤ Latency
➤ Number of
Servers
➤ Stability
➤ Cost of
RAM
➤ Cost of
RAM
➤ Scalability

OHIO
1)  No Hotspots
– DHT with RIPEMD160
simplifies data
partitioning
2)  Smart Client – 1 hop to
data, no load balancers
3)  Shared Nothing
Architecture,
every node identical
7) XDR – asynch replication
across data centers ensures
Zero Downtime
4)  Single row ACID
– synch replication in cluster
5)  Smart Cluster, Zero Touch
– auto-failover, rebalancing,
rolling upgrades..
6)  Transactions and long running
tasks prioritized real-time
Simpler Scaling: Fewer Servers, ACID, Zero Touch

Intelligent Client
•  Implements Aerospike API
•  Optimistic row locking
•  Optimized binary protocol
•  Cluster tracking
–  Learns about cluster
changes, partition map
–  Gossip protocol
•  Transaction semantics
–  Global transaction ID
–  Retransmit and timeout
Shields Applications from the Complexity of the Cluster

1.  Write sent to row master
2.  Latch against simultaneous writes
3.  Apply write synchronously to master memory
and replica memory
4.  Queue operations to disk
5.  Signal completed transaction
(optional storage commit wait)
6.  Master applies conflict resolution policy
(rollback/ rollforward)
master replica
1.  Cluster discovers new node via gossip
protocol
2.  Paxos vote determines new data
organization
3.  Partition migrations scheduled
4.  When a partition migration starts,
write journal starts on destination
5.  Partition moves atomically
6.  Journal is applied and source data deleted
transactions
continue
Writing with Immediate Consistency Adding a Node
ACID Transactions

➤  Distributed Hash Table with No Hotspots
§  Every key hashed with RIPEMD160
into a 20 byte (fixed length) string
NO KNOWN COLLISIONS
§  Hash + additional (fixed 64 bytes) data
stored in DRAM in the index
§  Some bits from hash value are used to
calculate the Partition ID (4096 partitions)
§  Partition ID maps to Node ID in the cluster
➤  1 Hop to data
§  Smart Client simply calculates Partition ID to
determine Node ID
§  No Load Balancers required
➤  Shared Nothing architecture
§  Every node is indentical
Distribution
cookie-abcdefg-12345678
182023kh15hh3kahdjsh
Partition ID Master
Node ID
Replica
Node ID
… 1 4
1820 2 3
1821 3 2
4096 4 1

➤  Super Storm Sandy 2012
§  NYC down for 17 hours
§  Back up and synched in 1 hour via
Aerospike Cross-Data Center Replication (XDR)
Replication that Works
“Aerospike allows us to
handle business continuity
and reliability across 4 data
centers seamlessly. And we
can now expand our
deployment to new data
centers in less than a week.”
- Elad Efraim, CTO

➤  Namespaces (policy containers)
§  Determine storage - DRAM or Flash
§  Determine replication factor
§  Contain records and sets
➤  Sets (tables) of records
§  Arbitrary grouping
➤  Records (rows)
§  Max 128k, contain key and bins
§  Bin with same name can contain
values of different types
u  String, integer, bytes (raw, blob, etc)
u  list ( an ordered collection of
values )
u  map ( a collection of keys and
values )
§  Bins can be added anytime
NOSQL EXTENSIBILITY

DISTRIBUTED QUERIES
1.  “Scatter” requests to all nodes
2.  Indexes in DRAM for fast map of secondary à primary keys
3.  Indexes co-located with data to guarantee ACID,
manage migrations
4.  Records read in parallel from all SSDs
using lock free concurrency control
5.  Aggregate results on each node
6.  “Gather” results from all nodes on client
STREAM AGGREGATIONS
1.  Push Code/ Security Policies/ Rules to Data with UDFs
2.  Pipe Query results through UDFs to
Filter, Transform, Aggregate.. Map, Reduce
REAL-TIME ANALYTICS on OPERATIONAL DATA (No ETL)
➤  In Database, within the same Cluster
➤  On the same Data, on XDR Replicated Clusters
Real-time Analytics on Operational Data

brian@aerospike.com
srini@aerospike.com
QUESTIONS

Storm Persistence and Real-Time Analytics

More Related Content

Storm Persistence and Real-Time Analytics