(STG202) AWS Import/Export Snowball: Large-Scale Data Ingest into AWS
•
4 likes•5,492 views
Moving terabyte and petabyte volumes of data into the cloud can be a challenge for many businesses. Come learn how you can use Snowball, a new AWS feature, to move large-scale (terabyte and petabyte) data to AWS storage services.
Report
Share
(STG202) AWS Import/Export Snowball: Large-Scale Data Ingest into AWS
- 1. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Colin Lazier, General Manager - AWS Import/Export & Amazon Glacier October 2015 Introduction to AWS Import/Export Snowball Accelerating Large-Scale Data Ingest into AWS Storage Services STG202
- 2. What to expect from the session • What is AWS Import/Export Snowball? • How does it work? • What are the two most commonly asked Snowball questions? • How fast is that truck full of drives? • How is my data transported securely? • Where can it go? • What does it cost? • Demo
- 3. What is Snowball? Petabyte scale data transport E-ink shipping label Ruggedized case “8.5G Impact” All data encrypted end-to-end 50 TB 10G network Rain & dust resistant Tamper-resistant case & electronics
- 4. How does it work?
- 5. Question 1:
- 6. Question 1: You have heard of networks, right?
- 7. Question 1: You have heard of networks, right? • Yes
- 8. Question 1: You have heard of networks, right? • Yes • However, for certain use cases networks are: Too slow Inelastic Too expensive
- 9. Question 1: You have heard of networks, right? • Yes • However, for certain use cases networks are: Too slow Inelastic Too expensive Bandwidth is growing more slowly than storage volumes
- 10. Question 1: You have heard of networks, right? • Snowball Fast Elastic Cost effective 10G network 50TB capacity
- 11. Question 1: You have heard of networks, right? • Snowball Fast Elastic Cost effective 10G network 50TB capacity
- 12. Question 1: You have heard of networks, right? • Snowball • Parallelize as needed 5X Snowballs 50G & 250TB
- 13. Question 2:
- 14. Question 2: Can I drop it?
- 15. Question 2: Can I drop it? • No (please don’t)
- 16. Question 2: Can I drop it? • No (please don’t) • Snowball is its own box • Has had many drop tests already • Can handle 8.5G impacts • Designed for shipping
- 17. Ok, let’s drop it
- 18. How fast is that truck full of drives? • Less than 1 day to transfer 50TB via a 10G connection with Snowball, less than 1 week including shipping • Number of days to transfer 50TB via the internet at typical utilizations Internet Connection Speed Utilization 1Gbps 500Mbps 300Mbps 150Mbps 25% 19 38 63 126 50% 9 19 32 63 75% 6 13 21 42
- 19. How fast is that truck full of drives? • Less than 1 day to transfer 250TB via 5x10G connections with 5 Snowballs, less than 1 week including shipping • Number of days to transfer 250TB via the Internet at typical utilizations Internet Connection Speed Utilization 1Gbps 500Mbps 300Mbps 150Mbps 25% 95 190 316 632 50% 47 95 158 316 75% 32 63 105 211
- 20. How is my data transported securely? • 256-bit encryption • Data is encrypted by the Snowball client before it reaches the Snowball appliance • Keys are managed by KMS and are never sent to the Snowball
- 21. How is my data transported securely? • Strong chain of custody • Tamper-resistant case • Tamper-resistant electronics (TPM) • Each Snowball is erased according to NIST 800-88 media sanitization guidelines between every job
- 22. From Where can it go? Import to Amazon S3, Export coming soon Amazon EMR Amazon Redshift Amazon S3 Amazon Glacier Amazon DynamoDBAmazon RDS Amazon EBS US-EAST-1 (Virginia) US-WEST-2 (Oregon)
- 23. What does it cost? • $200 / job plus shipping • Includes 10 days to fill the device at your site • $15/day after the tenth day on site • Standard Amazon S3 charges apply • $0.03/GB to transfer data out • $0.00/GB to transfer data in
- 24. What does it cost? Example 1: • 40TB loaded on to 1 Snowball • 2 days at your site • $200 plus shipping Example 2: • 30TB loaded on to 1 Snowball • 12 days at your site • $200 + 2*$15/day = $230 plus shipping
- 25. What does it cost? Example 3: • 140TB loaded on to 3 Snowballs • 8 days at your site • 3 * $200 = $600 plus shipping Example 4: • 30TB exported on to 1 Snowball • 8 days at your site • $200 + 30TB * $0.03/GB = $1,121.60 plus shipping
- 26. Demo
- 27. Overview
- 28. Step 1: Create a job
- 29. Step 1: Create a job
- 30. Step 1: Create a job
- 31. Step 1: Create a job
- 32. Step 2: Install client & await arrival
- 33. Step 3: Plug in your Snowball appliance
- 34. Step 4: Get your credentials
- 35. Step 5: Connect client to the Snowball appliance From a terminal: snowball start -i 192.0.2.0 -m /Downloads/JID2EXAMPLE-0c40-49a7-9f53- 916aEXAMPLE81-manifest.bin -u 12345-abcde-12345- ABCDE-123 snowball start -i [Snowball IP Address] -m [Path/to/manifest/file] -u [29 character unlock code]
- 36. Step 6: Copy your data From terminal: snowball cp --recursive /Logs/April s3://MyBucket/Logs/April snowball cp [options] [path/to/data/source] s3://[path/to/data/destination]
- 37. Step 7: Unplug your Snowball appliance & notify UPS
- 38. Step 7: Wait for import to complete
- 39. Thank you!
- 40. Remember to complete your evaluations!