Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
SlideShare a Scribd company logo

Tricks And Tradeoffs Of Deploying My Sql Clusters In The Cloud

•
•

1. MySQL databases can be deployed and managed in cloud computing environments like Amazon EC2 using tools that provide automation for launching slaves, backing up databases, and handling failover. 2. RightScale has been operating MySQL databases on Amazon EC2 since 2006 and provides a cloud management system and replicated MySQL product to automate the deployment and management of MySQL databases on EC2. 3. Some benefits of using cloud computing with databases include infinite computing resources, availability on demand with pay per use, and fully automatable database infrastructure management.

1 of 34
Download to read offline
MySQL in the Cloud Tricks and Tradeoffs Thorsten von Eicken CTO RightScale © 2009 RightScale Inc — All rights reserved 1
MySQL & Amazon EC2 @RightScale • Operating in Amazon EC2 since fall 2006 • Cloud Computing Management System • Replicated MySQL product offering mid ‘07 – 100’s of customer deployments – Automation to launch slaves, failover, backup, … • Animoto.com scaled to 4000 servers in 3.5 days © 2009 RightScale Inc — All rights reserved 2
Cloud Computing & Databases As many DB machines as 1. Infinite computing resources you want! 2. Available on-demand No cost for 3. Pay per use short periods 4. Delivered across the Internet Save work, More reliable v Fully Automatable Infrastructure © 2009 RightScale Inc — All rights reserved 3
Cloud Computing Vision • No up-front costs • Easy Dev & Test load clone time mon tue wed thu fri sat sun • Scale batch apps • Pay for average, not peak load peak load average load time mon tue wed thu fri sat sun © 2009 RightScale Inc — All rights reserved 4
Scale-up/down Automation • Scale-up www.mysite.com App server load > 50% – Majority voting – Apache Apache Add app servers – HAproxy HAproxy Tie into load bal & db – • Scale-down – App server load < 20% App Srvr App Srvr App Srvr App Srvr N/array 1/array – Majority voting MySQL MySQL replication master slave © 2009 RightScale Inc — All rights reserved 5
© 2009 RightScale Inc — All rights reserved 6
Load exceeds threshold Additional servers operational Additional servers terminated Load drops below threshold © 2009 RightScale Inc — All rights reserved 7
Starcut (Olympics) § Low cost to entry, scale up as demand grows § Repeatable, ”clean-room” design promotes stability and process § Great sandbox for testing and benchmarking © 2009 RightScale Inc — All rights reserved 8
Starcut (Stocks) Daily scale up/down to save costs § Global reach for services is needed (Europe, USA, Asia) § Big name clients demand carrier-grade reliability § Full 24/7, 365 support expectations to us and us to § our vendors © 2009 RightScale Inc — All rights reserved 9
Std MySQL/EBS configuration & Storage Array in the Sky © 2009 RightScale Inc — All rights reserved 10
Amazon EC2 Storage Architecture ½ core … 8 cores $0.10-$0.80/hr CPU 1.7GB … 16GB Memory 1 x 160GB … 4 x 420GB Local disks 1GB … 1TB volumes $0.10/GB/mo 1 … volumes / server Scalable $0.10/M-io-ops EBS disk array network attached, redundant redundant back-end store $0.15/GB/mo Persistent S3 Store S3 put/get semantics $0.10/K-ops © 2009 RightScale Inc — All rights reserved 11
Elastic Block Store (EBS) • Create volume: 1GB – 1TB size • Attach to any one instance in same availability zone • Pay $0.10/GB per month + $0.10/million I/O ops • Snapshot backup a volume to S3 • Create new volume from snapshot © 2009 RightScale Inc — All rights reserved 12
Snapshots Server 2 Server 3 Server 4 Server 1 • Point-in-time Vol 2 Vol 3 Vol 4 Vol 1 snapshot • Written to S3 – (Not to bucket!) snap 2 snap 1 Amazon S3 • Incremental snapshots • Restore to new volume – Instantaneous (lazy restore) © 2009 RightScale Inc — All rights reserved 13
Incremental Snapshots delete snap 2 • Table of Contents vs. Data Blocks • “Space used” difficult to gauge • Frequent snapshots: minimal cost © 2009 RightScale Inc — All rights reserved 14
Keeping Safe • Freeze data while snapshotting – XFS filesystem has freeze command – But: 2.6.18 kernel bug causes hung filesystem (ouch!) • Volumes are fragile in transit! – Unmounting can leave data in flight – Mounting mistake will be destructive • Use snapshots for safety! – Always snapshot after unmount – Create fresh volume from snapshot, mount that, retry in case of problem © 2009 RightScale Inc — All rights reserved 15
MySQL/EBS Replication App App Server 1 Server 2 MySQL MySQL replication master slave EBS EBS vol vol daily 10-minute snapshots snapshots Amazon Amazon S3 S3 © 2009 RightScale Inc — All rights reserved 16
Database operations pain points • Master hardware failure no replica ‘til hw fixed • Master software failure no replica ‘til root caused • Testing schema change extra system: 99% idle • Migrate to new hardware 2x rack space, SAN ports, … • New DB requirement machines, ports, disks, … • QA, test, stage, DR, demo, … always to few, always too idle © 2009 RightScale Inc — All rights reserved 17
Failure? Allocate fresh resources! MySQL MySQL MySQL MySQL MySQL dead replication master replication slave replication next replication next EBS EBS EBS EBS EBS vol vol vol vol vol snapshot snapshot snapshot snapshot snapshot init init Amazon Amazon Amazon Amazon Amazon S3 S3 S3 S3 S3 time • Extra resources are free for short periods! (e.g. 4-core/16GB + 1TB EBS => <$30/day © 2009 RightScale Inc — All rights reserved 18
Vertical Database Scaling Clone Server Config 1 www.rightscaledemo.com Load bal Load bal App Srvr App Srvr Small Small replication master slave clone config Large slave © 2009 RightScale Inc — All rights reserved 19
Vertical Database Scaling Clone Server Config 1 www.rightscaledemo.com Launch large slave 2 Load bal Load bal App Srvr App Srvr Small Small replication master slave replication Launch Large slave © 2009 RightScale Inc — All rights reserved 20
Vertical Database Scaling Clone Server 1 www.rightscaledemo.com Launch large slave 2 Load bal Load bal Promote to master 3 App Srvr App Srvr Small Small replication master slave replication master-db.rightscaledemo.com Large slave © 2009 RightScale Inc — All rights reserved 21
Vertical Database Scaling Clone Server 1 www.rightscaledemo.com Launch large slave 2 Load bal Load bal Promote to master 3 App Srvr App Srvr Launch new slave 4 Watch demo in Small Small replication master slave webinar recording at replication www.rightscale.com/webinars Launch Large Large replication master slave © 2009 RightScale Inc — All rights reserved 22
Cloud Principle #1: grab the next one! • Database management – Always roll forward, never fail-back – Animoto upgraded L -> XL @600 servers • App server management – Always roll forward, never fix in-place – www1 bust? Launch www3! Then troubleshoot • Good sysadmins have the urge to fix… L © 2009 RightScale Inc — All rights reserved 23
MySQL/EBS Toolbox server © 2009 RightScale Inc — All rights reserved 24
Many small databases Database pain cycle: 1. Recognize the DB is important 2. Buy expensive DB hardware 3. Consolidate DBs onto expensive hardware 4. Recognize the DB is now even more critical 5. Buy more expensive DB hardware 6. Repeat! Cloud Computing = Infinite servers and disks + Fully automatable © 2009 RightScale Inc — All rights reserved 25
Many Small Databases • Sharding – Split by user – Need to manage mappings • Read slaves – Many slave to increase read performance – Need to manage failover © 2009 RightScale Inc — All rights reserved 26
EBS Performance • Functionality (snapshots) trump performance • Very difficult to benchmark – Multiple levels of cache, some hidden • Sequential I/O limited by network – 70-90MB/sec typ, 100MB/sec max – 4 local drives striped yield more • Random I/O (sustained) – 400-1000 read ops/sec – 100-400 write ops/sec • Stripe to increase performance © 2009 RightScale Inc — All rights reserved 27
Clouds are Multiplying! – Amazon US + EU – GoGrid, Flexiscale – Eucalyptus open source release – RackSpace CloudServers – Sun cloud announcement – And many more… • Operating principles are the same – Storage architectures differ – From better local drives to traditional SANs © 2009 RightScale Inc — All rights reserved 28
Public vs. Private Cloud • Public cloud: Cloud Definition: – shared cloud computing 1. Infinite computing resources 2. Available on-demand infrastructure 3. Pay per use – connected to the public 4. Delivered across the Internet internet • Private cloud: – cloud infrastructure dedicated to a single party – may not be connected to the public internet • Hybrid cloud – private and public clouds used together – leverage the benefits of both © 2009 RightScale Inc — All rights reserved 29
Hybrid Cloud Use-Cases • Develop in public, deploy in private – Outsourced/distributed development – flexibility of test & dev resources • Develop in private, deploy in public – existing in house dev resources – ability to scale in public cloud – public-facing app • Private core, public expansion – Transaction intensive in-house core – Batch processing in public cloud © 2009 RightScale Inc — All rights reserved 30
RightScale Supports Hybrid Clouds 1. Commandeer some machines 2. Install Eucalyptus 3. Register with RightScale 4. The power of RightScale for your cloud 5. Invite friends to your cloud © 2009 RightScale Inc — All rights reserved 31
Multi-Cloud Architecture RightScale Core RackSpace EC2 GoGrid FlexiScale Eucalyptus Sun Cloud Controller more… Node Node Node Node Node Node © 2009 RightScale Inc — All rights reserved 32
Clouds Surpass Traditional Hosting Servers on-demand • Disk Volumes on-demand • Multiple datacenters • Reassign IPs / VIPs across datacenters • Multiple geographies • Multiple providers • • Automation tying it all together © 2009 RightScale Inc — All rights reserved 33
RightScale Cloud Management Platform Automation • Multi-Cloud • Fully transparent • Fully customizable • Replicated MySQL/EBS ServerTemplates • © 2009 RightScale Inc — All rights reserved 34

More Related Content

Tricks And Tradeoffs Of Deploying My Sql Clusters In The Cloud

  • 1. MySQL in the Cloud Tricks and Tradeoffs Thorsten von Eicken CTO RightScale © 2009 RightScale Inc — All rights reserved 1
  • 2. MySQL & Amazon EC2 @RightScale • Operating in Amazon EC2 since fall 2006 • Cloud Computing Management System • Replicated MySQL product offering mid ‘07 – 100’s of customer deployments – Automation to launch slaves, failover, backup, … • Animoto.com scaled to 4000 servers in 3.5 days © 2009 RightScale Inc — All rights reserved 2
  • 3. Cloud Computing & Databases As many DB machines as 1. Infinite computing resources you want! 2. Available on-demand No cost for 3. Pay per use short periods 4. Delivered across the Internet Save work, More reliable v Fully Automatable Infrastructure © 2009 RightScale Inc — All rights reserved 3
  • 4. Cloud Computing Vision • No up-front costs • Easy Dev & Test load clone time mon tue wed thu fri sat sun • Scale batch apps • Pay for average, not peak load peak load average load time mon tue wed thu fri sat sun © 2009 RightScale Inc — All rights reserved 4
  • 5. Scale-up/down Automation • Scale-up www.mysite.com App server load > 50% – Majority voting – Apache Apache Add app servers – HAproxy HAproxy Tie into load bal & db – • Scale-down – App server load < 20% App Srvr App Srvr App Srvr App Srvr N/array 1/array – Majority voting MySQL MySQL replication master slave © 2009 RightScale Inc — All rights reserved 5
  • 6. © 2009 RightScale Inc — All rights reserved 6
  • 7. Load exceeds threshold Additional servers operational Additional servers terminated Load drops below threshold © 2009 RightScale Inc — All rights reserved 7
  • 8. Starcut (Olympics) § Low cost to entry, scale up as demand grows § Repeatable, ”clean-room” design promotes stability and process § Great sandbox for testing and benchmarking © 2009 RightScale Inc — All rights reserved 8
  • 9. Starcut (Stocks) Daily scale up/down to save costs § Global reach for services is needed (Europe, USA, Asia) § Big name clients demand carrier-grade reliability § Full 24/7, 365 support expectations to us and us to § our vendors © 2009 RightScale Inc — All rights reserved 9
  • 10. Std MySQL/EBS configuration & Storage Array in the Sky © 2009 RightScale Inc — All rights reserved 10
  • 11. Amazon EC2 Storage Architecture ½ core … 8 cores $0.10-$0.80/hr CPU 1.7GB … 16GB Memory 1 x 160GB … 4 x 420GB Local disks 1GB … 1TB volumes $0.10/GB/mo 1 … volumes / server Scalable $0.10/M-io-ops EBS disk array network attached, redundant redundant back-end store $0.15/GB/mo Persistent S3 Store S3 put/get semantics $0.10/K-ops © 2009 RightScale Inc — All rights reserved 11
  • 12. Elastic Block Store (EBS) • Create volume: 1GB – 1TB size • Attach to any one instance in same availability zone • Pay $0.10/GB per month + $0.10/million I/O ops • Snapshot backup a volume to S3 • Create new volume from snapshot © 2009 RightScale Inc — All rights reserved 12
  • 13. Snapshots Server 2 Server 3 Server 4 Server 1 • Point-in-time Vol 2 Vol 3 Vol 4 Vol 1 snapshot • Written to S3 – (Not to bucket!) snap 2 snap 1 Amazon S3 • Incremental snapshots • Restore to new volume – Instantaneous (lazy restore) © 2009 RightScale Inc — All rights reserved 13
  • 14. Incremental Snapshots delete snap 2 • Table of Contents vs. Data Blocks • “Space used” difficult to gauge • Frequent snapshots: minimal cost © 2009 RightScale Inc — All rights reserved 14
  • 15. Keeping Safe • Freeze data while snapshotting – XFS filesystem has freeze command – But: 2.6.18 kernel bug causes hung filesystem (ouch!) • Volumes are fragile in transit! – Unmounting can leave data in flight – Mounting mistake will be destructive • Use snapshots for safety! – Always snapshot after unmount – Create fresh volume from snapshot, mount that, retry in case of problem © 2009 RightScale Inc — All rights reserved 15
  • 16. MySQL/EBS Replication App App Server 1 Server 2 MySQL MySQL replication master slave EBS EBS vol vol daily 10-minute snapshots snapshots Amazon Amazon S3 S3 © 2009 RightScale Inc — All rights reserved 16
  • 17. Database operations pain points • Master hardware failure no replica ‘til hw fixed • Master software failure no replica ‘til root caused • Testing schema change extra system: 99% idle • Migrate to new hardware 2x rack space, SAN ports, … • New DB requirement machines, ports, disks, … • QA, test, stage, DR, demo, … always to few, always too idle © 2009 RightScale Inc — All rights reserved 17
  • 18. Failure? Allocate fresh resources! MySQL MySQL MySQL MySQL MySQL dead replication master replication slave replication next replication next EBS EBS EBS EBS EBS vol vol vol vol vol snapshot snapshot snapshot snapshot snapshot init init Amazon Amazon Amazon Amazon Amazon S3 S3 S3 S3 S3 time • Extra resources are free for short periods! (e.g. 4-core/16GB + 1TB EBS => <$30/day © 2009 RightScale Inc — All rights reserved 18
  • 19. Vertical Database Scaling Clone Server Config 1 www.rightscaledemo.com Load bal Load bal App Srvr App Srvr Small Small replication master slave clone config Large slave © 2009 RightScale Inc — All rights reserved 19
  • 20. Vertical Database Scaling Clone Server Config 1 www.rightscaledemo.com Launch large slave 2 Load bal Load bal App Srvr App Srvr Small Small replication master slave replication Launch Large slave © 2009 RightScale Inc — All rights reserved 20
  • 21. Vertical Database Scaling Clone Server 1 www.rightscaledemo.com Launch large slave 2 Load bal Load bal Promote to master 3 App Srvr App Srvr Small Small replication master slave replication master-db.rightscaledemo.com Large slave © 2009 RightScale Inc — All rights reserved 21
  • 22. Vertical Database Scaling Clone Server 1 www.rightscaledemo.com Launch large slave 2 Load bal Load bal Promote to master 3 App Srvr App Srvr Launch new slave 4 Watch demo in Small Small replication master slave webinar recording at replication www.rightscale.com/webinars Launch Large Large replication master slave © 2009 RightScale Inc — All rights reserved 22
  • 23. Cloud Principle #1: grab the next one! • Database management – Always roll forward, never fail-back – Animoto upgraded L -> XL @600 servers • App server management – Always roll forward, never fix in-place – www1 bust? Launch www3! Then troubleshoot • Good sysadmins have the urge to fix… L © 2009 RightScale Inc — All rights reserved 23
  • 24. MySQL/EBS Toolbox server © 2009 RightScale Inc — All rights reserved 24
  • 25. Many small databases Database pain cycle: 1. Recognize the DB is important 2. Buy expensive DB hardware 3. Consolidate DBs onto expensive hardware 4. Recognize the DB is now even more critical 5. Buy more expensive DB hardware 6. Repeat! Cloud Computing = Infinite servers and disks + Fully automatable © 2009 RightScale Inc — All rights reserved 25
  • 26. Many Small Databases • Sharding – Split by user – Need to manage mappings • Read slaves – Many slave to increase read performance – Need to manage failover © 2009 RightScale Inc — All rights reserved 26
  • 27. EBS Performance • Functionality (snapshots) trump performance • Very difficult to benchmark – Multiple levels of cache, some hidden • Sequential I/O limited by network – 70-90MB/sec typ, 100MB/sec max – 4 local drives striped yield more • Random I/O (sustained) – 400-1000 read ops/sec – 100-400 write ops/sec • Stripe to increase performance © 2009 RightScale Inc — All rights reserved 27
  • 28. Clouds are Multiplying! – Amazon US + EU – GoGrid, Flexiscale – Eucalyptus open source release – RackSpace CloudServers – Sun cloud announcement – And many more… • Operating principles are the same – Storage architectures differ – From better local drives to traditional SANs © 2009 RightScale Inc — All rights reserved 28
  • 29. Public vs. Private Cloud • Public cloud: Cloud Definition: – shared cloud computing 1. Infinite computing resources 2. Available on-demand infrastructure 3. Pay per use – connected to the public 4. Delivered across the Internet internet • Private cloud: – cloud infrastructure dedicated to a single party – may not be connected to the public internet • Hybrid cloud – private and public clouds used together – leverage the benefits of both © 2009 RightScale Inc — All rights reserved 29
  • 30. Hybrid Cloud Use-Cases • Develop in public, deploy in private – Outsourced/distributed development – flexibility of test & dev resources • Develop in private, deploy in public – existing in house dev resources – ability to scale in public cloud – public-facing app • Private core, public expansion – Transaction intensive in-house core – Batch processing in public cloud © 2009 RightScale Inc — All rights reserved 30
  • 31. RightScale Supports Hybrid Clouds 1. Commandeer some machines 2. Install Eucalyptus 3. Register with RightScale 4. The power of RightScale for your cloud 5. Invite friends to your cloud © 2009 RightScale Inc — All rights reserved 31
  • 32. Multi-Cloud Architecture RightScale Core RackSpace EC2 GoGrid FlexiScale Eucalyptus Sun Cloud Controller more… Node Node Node Node Node Node © 2009 RightScale Inc — All rights reserved 32
  • 33. Clouds Surpass Traditional Hosting Servers on-demand • Disk Volumes on-demand • Multiple datacenters • Reassign IPs / VIPs across datacenters • Multiple geographies • Multiple providers • • Automation tying it all together © 2009 RightScale Inc — All rights reserved 33
  • 34. RightScale Cloud Management Platform Automation • Multi-Cloud • Fully transparent • Fully customizable • Replicated MySQL/EBS ServerTemplates • © 2009 RightScale Inc — All rights reserved 34