Practical Distributed Processing

Using MySQL Built-In Functionality

Bob Burgess radian6 Technologies MySQL User Conference 2010

Who Am I
Database Analyst, radian6 Several Years Experience:

Sybase, Oracle, MySQL AIX, Linux programming

Scope
MySQL concepts used in this system How I solved our particular problem Practical set-up of our distributed system

Scope NOT
Complete course on distributed processing Complete coverage of related MySQL functionality

The Problem
Influencer calculation: complex and getting more complex More and more data More and more customers and profiles Not scalable to run the stored procedure on just the main database

Design Goals
Move calculation off of main database Keep existing stored procedure No single controller Horizontally scalable!

Influencer calculation
Whos most influential? (For this topic.)

subset of database do stuff influencer data

(two MyISAM tables)

sphinx index

MySQL Functionality Used

MySQL Functionality Used

Replication Blackhole Storage Engine Federated Storage Engine Public-Key Authentication

Replication
Master Database Replica Database

binlogs
INSERT... UPDATE... INSERT... DELETE...

relay logs
INSERT... UPDATE... INSERT... DELETE...

Replication
Master Database
server_id = 1 log_bin = /store/log/mysql-bin grant replication slave on *.* to 'repl'@'%' identified by 'secret'; /store/log/mysql-bin.000001

binlogs
INSERT... UPDATE... INSERT... DELETE...

Replication
server_id = 2 relaylog = /store/relaylog/relaylog
change master to master_host="masterserver", master_port=3306, master_user="repl", master_password="secret", master_log_file="mysql-bin.000001", master_log_pos=0; start slave;

Replica Database

relay logs
INSERT... UPDATE... INSERT... DELETE...

/store/relaylog/relaylog.000001

Black Hole Tables

> create table t (a int) engine=blackhole; Query OK, 0 rows affected (0.02 sec)
> insert into t values (1); Query OK, 1 row affected (0.00 sec) > select * from t; Empty set (0.00 sec)

Black Hole Tables

Before-Insert trigger

create trigger t_bi before insert on t for each row insert into real_table values (new.a);

real table

Federated Tables

Points to a table in another database

server1

table `t`

create table t (a int)

federated table `tm` server2 create table tm (a int) engine=federated connection="mysql://user:pass@server1/schemaname/t"

Not enabled by default in 5.1!

Federated Tables

Points to a table in another database

server1

table `t`

create table t (a int)

federated table `tm` server2

federated table `tm` server3

federated table `tm` server4

create table tm (a int) engine=federated connection="mysql://user:pass@server1/schemaname/t"

Public-Key Authentication

Secure shell (ssh) or secure copy (scp) without a password

Public-Key Authentication

# scp file mysql@main:/...

worker

main db

# cd /root/.ssh # ssh-keygen # cat id_rsa.pub

# vi /home/mysql/.ssh/authorized_keys

Distributed System Why It Works For Us

All involved database data fits on the worker database Calculation produces a self-contained data set MyISAM can be used for the resulting data

Distributed System General Architecture

main db server main db
replication
federated
MYD & MYI

scp file copy

worker db
worker 1

MYD & MYI

Distributed System General Architecture

main db server main db
MYD & MYI MYD & MYI

MYD & MYI

worker db worker 1

MYD & MYI

worker db worker 3 worker db worker 2

MYD & MYI

MYD & MYI

The Non-Queue... Queue

--- Topics --No. Name 101 Apples 102 Oranges 103 Grapes 104 Bananas 105 Pears

--- To-do List --101 104 105

The Non-Queue... Queue

------ Topics--Topics --101 Apples 102 Oranges No. Name 103 Grapes 101 Apples 104 Bananas 102 Oranges 105 Pears

Recalc -Reqd Full No 103 Grapes No 104 Bananas Partial 105 Pears Full

Recalc -Date Priority Running 10:00am high 101 8:00am 0 7:30am 0 10:10am high 302 10:05am low 0

not running highest priority earliest recalc date

Processing Cycle
Get work... Get next Topic to do

is there any? yes

no

sleep

Processing Cycle
Is data current? Let replication catch up!

replication stopped?

yes

sleep 10 min.

replication behind?

yes

sleep 1 min.

Processing Cycle
Replace stale Topic Profile someone else likely has it...

waited for replication?

yes

get a new Topic to do

Processing Cycle
Playing Nice with Others
mark Topic with my ID

sleep short random time

Topic still has my ID?

no

sleep 1 min.

begin again

Processing Cycle
run the stored proc

success ? yes

no

log the error

sleep 1 min.

begin again

Processing Cycle
flush the data table for this Profile (locally)

copy to main db server

flush the data table for this Profile (on main db)

update Profile table on main: not running

Processing Cycle

back to the top

Code main loop

trap 'STOP=1' TERM STOP=0 while [ $STOP -eq 0 ] ; do

done

Code who am I?
NODEID_QUERY="select @@server_id-1000;" INSTANCE=$2 NODEID=`$MYSQL -e"$NODEID_QUERY"` UNIQUE_INSTANCEID=$((NODEID*100+INSTANCE))

Code whos next?

NEEDY=`$MYSQL_MASTER -e"$NEEDY_TP_QUERY" \ 2>/tmp/mysql.err` set -- $NEEDY TPID=$1; RECALC_SCOPE=$2; RECALC_DATE=$3

Code whos next, the SQL

NEEDY_TP_QUERY=" select topicFilterId,influencerRecalcReqd, unix_timestamp(influencerRecalcDate) RecalcDate from TopicFilter where influencerRecalcReqd>0 and influencerRunning=0 and influencerRecalcDate<=now() and active in (1,2) and influencerPriority>=$MINPRIORITY order by influencerPriority desc, influencerRecalcDate asc, topicFilterId limit 1;"

Code inner loop

while [ ! -z "${TPID}" -a $STOP -eq 0 ]; do

done

Code how far behind?

REPLICATION_DELAY= `$MYSQL -e'show slave status\G' | grep "Seconds_Behind_Master" | cut -f2 -d':'`

Code call the proc

$MYSQL -e"call proc(${TPID},${RECALC_SCOPE})" >${INFLUENCER_LOG}${TPID}.${INSTANCE}.log 2>&1 RUN_SUCCESS=`tail -1 ${INFLUENCER_LOG}${TPID}.${INSTANCE}.log | grep -c SUMMARY` if [ $RUN_SUCCESS -eq 1 ] ; then copy tables else report error fi

Code send the result back

$MYSQL -e"use influencer; flush tables Dynamics_${TPID};" if scp -p -B ${TABLE_SOURCE}/Dynamics_${TPID}.* ${TABLE_MASTER_PUSH_LOC} 2>&1 && \ scp -p -B ${TABLE_SOURCE}/Post_${TPID}.* ${TABLE_MASTER_PUSH_LOC} 2>&1 then copy files to backup remove tables from worker else report error fi

Code was I killed?

if [ $STOP -eq 1 ]; then echo "---- Script stopped by Kill (TERM signal) ----" fi

A Blackhole Table Trigger

create table BlogPost (...) engine=blackhole; create table PostAuthor ( blogPostId bigint not null primary key, author varchar(189)) default character set=utf8; create trigger BlogPost_bir before insert on BlogPost for each row begin if new.mediaProviderId>1 then insert ignore into PostAuthor (blogPostId,author) values (new.blogPostId, new.author); end if; end;

Launch Script
Shell script to launch workers On

Starts several workers

Off
sends a signal to each worker stops after this stored proc call

Kill
abort and clean up database

Monitor Scripts
Replication Disk space Errors from the worker script

Procedure fails scp fails gzip fails replication is behind

Design Decisions

MyISAM vs. InnoDB

Size Tables easily copied Table locking / Replication: Smaller Queries Required

SSD Storage

Development Directions
Migrate stored proc to Java More workers Problems in Code not atomic:

flush table / copy table copy result tables to main db

Take Away...
Replication and Federated tables aren't hard. Thousands of MyISAM tables isn't crazy.

Distributed

processing isn't rocket surgery.

Thank you!
Bob Burgess bob.burgess@radian6.com

Slides on the conference site. Email me for sample scripts.