Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
SlideShare a Scribd company logo

High Performance MongoDB Clusters with Amazon EBS Provisioned IOPS

•
•
Amazon Web Services
Amazon Web Services

This document discusses high performance MongoDB clusters on AWS using Amazon EBS Provisioned IOPS volumes. It covers AWS storage options, introduces Provisioned IOPS volumes for consistent I/O performance, discusses best practices for running MongoDB on AWS including using Provisioned IOPS volumes, and how Parse runs MongoDB on AWS and benefits from Provisioned IOPS.

1 of 42
High Performance MongoDB Clusters with Amazon EBS Provisioned IOPS Jafar Shameem Business Development Manager, Amazon Web Services Sandeep Parikh Solutions Architect, 10gen Miles Ward Solutions Architect, Amazon Web Services Charity Majors Engineer, Parse
Agenda • AWS Storage Options and EBS • Introducing Provisioned IOPS • About MongoDB • MongoDB on AWS best practices • How Parse runs MongoDB on AWS and benefits from Provisioned IOPS
Storage Options on AWS Block Storage Object Storage (Elastic Block Store) (S3, Glacier) Use for: Use for: • Access to raw • Pictures, videos, unformatted block highly durable level storage media storage • Persistent Storage • Cold storage for long-term archive
Amazon Elastic Block Store (EBS) Elastic Block Storage: Persistent Storage for EC2 Feature Details High Mount EBS as drives and performance format as required file system Flexible size Volumes from 1GB to 1TB in size Secure Private to your instances Available Replicated within an High performance block storage Availability Zone device Backups Volumes can be snapshotted Mount as drives to instances for point in time restore Persistent and independent of Monitoring Detailed metrics captured via instance lifecycle Cloud Watch
Introducing Provisioned IOPS Volumes
Introducing Provisioned IOPS Volumes ❶ Select a new type of Provisioned IOPS volumes ❷ Specify the volume capacity ❸ Specify the number of IOs per second your application needs, up to 2000 PIOPS per volume. The volume will deliver the specified IO per second. $ ec2-create-volume --size 500 --availability-zone us-east-1b --type io1 –iops 2000
When should I use Provisioned IOPS? • Standard volumes are designed for boot volumes, running applications with moderate and bursty IO, sequential IO access • Provisioned IOPS volumes are designed for running transactional applications that require high and consistent IO: o Relational Databases o NoSQL Databases, e.g. MongoDB o Productivity applications, e.g. Microsoft Exchange o Enterprise Applications
#aws MongoDB and AWS Sandeep Parikh Solutions Architect, 10gen
Database Landscape
MongoDB Basics • Open source • Document database • High performance • Horizontally scalable • Fully featured • Built to match agile development and deployment
MongoDB Features • Flexible document data model • Rich ad-hoc queries • Real-time aggregation • Support for geospatial data and queries • Built-in scalability – Redundancy, failover and auto-partitioning • Broad programming language support
Relational Model
MongoDB Data Model
Content Operational High Volume User Data E-Commerce Management Intelligence Data Feeds Management Use Cases
Scaling MongoDB • Replica Sets – Redundancy, failover, read scalability • Sharding – Auto-partitions data, read/write scalability • Multi-datacenter deployments • Tunable consistency • Engineering for zero downtime
Replica Sets
Sharding
Mapping MongoDB to AWS • MongoDB scales easily across multiple nodes • High memory and fast storage • Durability and performance • Deploy across multiple data centers • Quickly add capacity
High Performance Workloads • Instances support up to 240+ GiB memory • Provisioned IOPS provides consistent and predictable EBS performance • RAID10 maximizes performance and durability • EBS snapshots integrate with MongoDB – Fast and easy backup – Quickly restore and deploy additional nodes
MongoDB on AWS Resources • Best practices for configuring MongoDB • MongoDB on AWS white paper • Sample CloudFormation templates to launch – Single node deployments – Pre-configured multi-node replica sets
MongoDB Monitoring Service
MongoDB on AWS Best Practices
Scaling MongoDB over time
High Performance MongoDB Clusters with Amazon EBS Provisioned IOPS
High Performance MongoDB Clusters with Amazon EBS Provisioned IOPS
High Performance MongoDB Clusters with Amazon EBS Provisioned IOPS
High Performance MongoDB Clusters with Amazon EBS Provisioned IOPS
IO Considerations MongoDB IO Workloads • Journal • Background flushing • Page Faults EBS Optimized RAID 10
Extra big scale
Backup MongoDB using low priority replica
Testing random 4K reads • One Volume: ~200 MongoOPS with some variability, <1mb/s EBS • Loaded instance: ~ 1000 MongoOPS with some variability <10mb/s + PIOPS • One Volume: 2000 MongoOPS with <1% variability, 3mb/s • Loaded Instance: 20000 MongoOPS with <1% variability, 60mb/s • Hi1.4xlarge ephemeral: ~64,000 MongoOPS with low variability, SSD ~245mb/s
Testing random 4K reads EBS + PIOPS + PIOPS SSD Stable Stable
Stability Tips • Ext4 or XFS • Reduce I/O overhead – • nodiratime, noatime • Raise file descriptor limits • Set disk read-ahead • No large virtual memory pages • SNAPSHOT SNAPSHOT SNAPSHOT
High Performance MongoDB Clusters with Amazon EBS Provisioned IOPS
Parse runs on AWS • We provide a platform for mobile developers • SDKs and APIs for iOS, Android, Windows mobile apps -- 60,000 and counting • Highly elastic traffic patterns • As a platform, we can’t go down, so cloud computing is the only way to go
Why we use MongoDB • Mongo is not optimized for one task, it’s very good across a broad range of tasks • Schema-less is a must • We can do auto-indexing based on an analysis of your traffic and data • Horizontal scalability • Reliability across multiple Availability Zones
The ways we use MongoDB • User data • RAID 0 striped P-IOPS 1000 EBS volumes • smart indexing, EC2 snapshots, continuous compaction on snapshot nodes • sharded in the application layer • Common data • Real-time query analysis • ephemeral storage for disposable data • Billing and logging
Things we’ve learned • Always use replica sets. Spread across AZs. • Chef everything so nodes are disposable • Warm up your secondaries before promoting. • Snapshot often. Never allow your snapshot host to become primary • set priority=0, also hidden=true if you read off secondaries • Use Provisioned IOPS and ebs_optimized if you care about your data
End-to-end latency Before PIOPS: After PIOPS:
In summary • We love MongoDB. The rate of development and improvement is incredible. • PIOPS made it feasible for us to stay on AWS • Parse wouldn’t be possible without both the elasticity and power of AWS and the functionality of MongoDB
High Performance MongoDB Clusters with Amazon EBS Provisioned IOPS
Get started on Provisioned IOPS today! aws.amazon.com/ebs Questions: e-mail: shameemj@amazon.com • MongoDB CloudFormation Templates: o http://docs.mongodb.org/ecosystem/tutorial/automate-deployment-with-cloudformation/ • MongoDB on AWS: o http://docs.mongodb.org/ecosystem/platforms/#amazon-web-services-ec2 • Slides will be made available here: o http://aws.amazon.com/ebs/webinars/ • Benchmarking EBS performance: o http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSPerformance.html

More Related Content

High Performance MongoDB Clusters with Amazon EBS Provisioned IOPS

  • 1. High Performance MongoDB Clusters with Amazon EBS Provisioned IOPS Jafar Shameem Business Development Manager, Amazon Web Services Sandeep Parikh Solutions Architect, 10gen Miles Ward Solutions Architect, Amazon Web Services Charity Majors Engineer, Parse
  • 2. Agenda • AWS Storage Options and EBS • Introducing Provisioned IOPS • About MongoDB • MongoDB on AWS best practices • How Parse runs MongoDB on AWS and benefits from Provisioned IOPS
  • 3. Storage Options on AWS Block Storage Object Storage (Elastic Block Store) (S3, Glacier) Use for: Use for: • Access to raw • Pictures, videos, unformatted block highly durable level storage media storage • Persistent Storage • Cold storage for long-term archive
  • 4. Amazon Elastic Block Store (EBS) Elastic Block Storage: Persistent Storage for EC2 Feature Details High Mount EBS as drives and performance format as required file system Flexible size Volumes from 1GB to 1TB in size Secure Private to your instances Available Replicated within an High performance block storage Availability Zone device Backups Volumes can be snapshotted Mount as drives to instances for point in time restore Persistent and independent of Monitoring Detailed metrics captured via instance lifecycle Cloud Watch
  • 6. Introducing Provisioned IOPS Volumes ❶ Select a new type of Provisioned IOPS volumes ❷ Specify the volume capacity ❸ Specify the number of IOs per second your application needs, up to 2000 PIOPS per volume. The volume will deliver the specified IO per second. $ ec2-create-volume --size 500 --availability-zone us-east-1b --type io1 –iops 2000
  • 7. When should I use Provisioned IOPS? • Standard volumes are designed for boot volumes, running applications with moderate and bursty IO, sequential IO access • Provisioned IOPS volumes are designed for running transactional applications that require high and consistent IO: o Relational Databases o NoSQL Databases, e.g. MongoDB o Productivity applications, e.g. Microsoft Exchange o Enterprise Applications
  • 8. #aws MongoDB and AWS Sandeep Parikh Solutions Architect, 10gen
  • 10. MongoDB Basics • Open source • Document database • High performance • Horizontally scalable • Fully featured • Built to match agile development and deployment
  • 11. MongoDB Features • Flexible document data model • Rich ad-hoc queries • Real-time aggregation • Support for geospatial data and queries • Built-in scalability – Redundancy, failover and auto-partitioning • Broad programming language support
  • 14. Content Operational High Volume User Data E-Commerce Management Intelligence Data Feeds Management Use Cases
  • 15. Scaling MongoDB • Replica Sets – Redundancy, failover, read scalability • Sharding – Auto-partitions data, read/write scalability • Multi-datacenter deployments • Tunable consistency • Engineering for zero downtime
  • 18. Mapping MongoDB to AWS • MongoDB scales easily across multiple nodes • High memory and fast storage • Durability and performance • Deploy across multiple data centers • Quickly add capacity
  • 19. High Performance Workloads • Instances support up to 240+ GiB memory • Provisioned IOPS provides consistent and predictable EBS performance • RAID10 maximizes performance and durability • EBS snapshots integrate with MongoDB – Fast and easy backup – Quickly restore and deploy additional nodes
  • 20. MongoDB on AWS Resources • Best practices for configuring MongoDB • MongoDB on AWS white paper • Sample CloudFormation templates to launch – Single node deployments – Pre-configured multi-node replica sets
  • 22. MongoDB on AWS Best Practices
  • 28. IO Considerations MongoDB IO Workloads • Journal • Background flushing • Page Faults EBS Optimized RAID 10
  • 30. Backup MongoDB using low priority replica
  • 31. Testing random 4K reads • One Volume: ~200 MongoOPS with some variability, <1mb/s EBS • Loaded instance: ~ 1000 MongoOPS with some variability <10mb/s + PIOPS • One Volume: 2000 MongoOPS with <1% variability, 3mb/s • Loaded Instance: 20000 MongoOPS with <1% variability, 60mb/s • Hi1.4xlarge ephemeral: ~64,000 MongoOPS with low variability, SSD ~245mb/s
  • 32. Testing random 4K reads EBS + PIOPS + PIOPS SSD Stable Stable
  • 33. Stability Tips • Ext4 or XFS • Reduce I/O overhead – • nodiratime, noatime • Raise file descriptor limits • Set disk read-ahead • No large virtual memory pages • SNAPSHOT SNAPSHOT SNAPSHOT
  • 35. Parse runs on AWS • We provide a platform for mobile developers • SDKs and APIs for iOS, Android, Windows mobile apps -- 60,000 and counting • Highly elastic traffic patterns • As a platform, we can’t go down, so cloud computing is the only way to go
  • 36. Why we use MongoDB • Mongo is not optimized for one task, it’s very good across a broad range of tasks • Schema-less is a must • We can do auto-indexing based on an analysis of your traffic and data • Horizontal scalability • Reliability across multiple Availability Zones
  • 37. The ways we use MongoDB • User data • RAID 0 striped P-IOPS 1000 EBS volumes • smart indexing, EC2 snapshots, continuous compaction on snapshot nodes • sharded in the application layer • Common data • Real-time query analysis • ephemeral storage for disposable data • Billing and logging
  • 38. Things we’ve learned • Always use replica sets. Spread across AZs. • Chef everything so nodes are disposable • Warm up your secondaries before promoting. • Snapshot often. Never allow your snapshot host to become primary • set priority=0, also hidden=true if you read off secondaries • Use Provisioned IOPS and ebs_optimized if you care about your data
  • 40. In summary • We love MongoDB. The rate of development and improvement is incredible. • PIOPS made it feasible for us to stay on AWS • Parse wouldn’t be possible without both the elasticity and power of AWS and the functionality of MongoDB
  • 42. Get started on Provisioned IOPS today! aws.amazon.com/ebs Questions: e-mail: shameemj@amazon.com • MongoDB CloudFormation Templates: o http://docs.mongodb.org/ecosystem/tutorial/automate-deployment-with-cloudformation/ • MongoDB on AWS: o http://docs.mongodb.org/ecosystem/platforms/#amazon-web-services-ec2 • Slides will be made available here: o http://aws.amazon.com/ebs/webinars/ • Benchmarking EBS performance: o http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSPerformance.html

Editor's Notes

  1. We will post these slides on the EBS website soon.