Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
SlideShare a Scribd company logo

Pstrong Cybera 29 Sept 2008

•
•
Cybera Inc.
Cybera Inc.

This document summarizes an eBay presentation on next generation data centers and cloud computing technologies. It provides examples of how eBay has leveraged these technologies, including virtualizing and scaling their database to handle extreme growth. The presentation discusses how next generation data centers are more than just technologies, and focus on running business processes driven by service level agreements.

1 of 38
eBay – Examples Of NGDC Technologies At Work Paul Strong Distinguished Research Scientist, eBay Research Labs ® Cybera Summit 2008
Copyright Notice © 2008 eBay Inc. All rights reserved. • No part of these materials may be reproduced or transmitted in any form, by any means (electronic, photocopying, recording, or otherwise) without the prior permission of eBay Inc. • eBay and the eBay logo are registered trademarks of eBay Inc. • PayPal and the PayPal logo are registered trademarks of PayPal, Inc. • Other trademarks and brands are the property of their respective owners. • Please do not take our picture or record the class/session without asking permission. ® ©2008, eBay Inc.
What Is A Next Generation Data Center? • Is it the platform for SOA? • Is it a Grid? • Does it leverage Virtualization technologies? • Is it full of blades? • Is it Greener? • Is it more efficient? • Is it more automated? ® ©2008, eBay Inc.
What Is A Next Generation Data Center? • Is it the platform for SOA? • Is it a Grid? • Does it leverage Virtualization technologies? • Is it full of blades? • Is it Greener? • Is it more efficient? • Is it more automated? ® ©2008, eBay Inc.
NGDC Is… More Than A Collection Of Technologies • Runs Business Processes driven by SLAs • Is a Value Center, rather than a Cost Center – A new dialog between IT and the Business • Enables – Internal utilities for core business functions/processes – External utilities for business process elements – Businesses based on business process mash-ups – Opportunities for new platforms ® ©2008, eBay Inc.
Cloud Computing ® ©2008, eBay Inc.
Cloud Computing Is… • A new way about thinking of business… – Business = Codified core differentiating value • Outsourcing everything that is not core – Physical infrastructure → Clouds, Platform as a Service (PaaS) – Logical infrastructure (non-core business software) → Software as a Service (SaaS) • About being flexible – Temporary outsourcing… • Not a new set of technologies – A new context for applying a set of evolving technologies that are now more mature or capable ® ©2008, eBay Inc.
Cloud Computing Is About… ® ©2008, eBay Inc.
NGDC Is… More Than A Collection Of Technologies • Runs Business Processes driven by SLAs • Is a Value Center, rather than a Cost Center – A new dialog between IT and the Business • Enables – Internal utilities clouds for core business functions/processes – External utilities clouds (PaaS, SaaS) for infrastructure and business process elements – Businesses based on business process mash-ups – Opportunities for new platforms ® ©2008, eBay Inc.
NGDC, Friends & Relations U ® ©2008, eBay Inc.
Managing NGDCs… Is not about managing servers, operating systems, disks etc. ® ©2008, eBay Inc.
® ©2008, eBay Inc.
Why eBay Is A Useful Example New Challenges Extreme Engineering The Bleeding Edge Everyday use Technology trickle down/transfer ® ©2008, eBay Inc.
eBay – The 30 Second Introduction! eBay users trade about $1,989 worth of goods on the site every second On an average day on eBay… • A car sells every 56 seconds • A pair of shoes sell every 7 seconds • A cell phones sells every 7 seconds 1.3m people make all or part of their living selling on * ® ©2008, eBay Inc. *ACNielsen International Research, June 2006
eBay’s Drivers • Extreme Scale – 241m Registers Users, 112m+ Items, 7m+ New Items Per Day • Extreme Growth – Near exponential growth in listings for most of history – 13 years • Extreme Agility – Roll code to the site every 2 weeks • Constant, predictable presence – Must be 24x7x365 • Efficiency Failure To Keep Up Is Not An Option! ® ©2008, eBay Inc.
eBay – The Simple Diagram ® ©2008, eBay Inc.
eBay Example #1 Making The Database Scale • Second Database for failover • CGI pools, Listings, Pages, and Search continued to scale horizontally However … By November 1999, the database servers approached their limits of physical growth. S/W Load Balancer S/W Load Balancer S/W Load Balancer S/W Load Balancer Web Server Web Server Web Server Apache C++ COTS Search OS OS OS UNIX “CGIn” “Listings” “Pages” “Search” RDBMS RDBMS UNIX UNIX 1999 bull.ebay.com bear.ebay.com ® ©2008, eBay Inc.
eBay Example #1 Making The Database Scale • Database quot;splitquot; technology. • Logically partition database into separate instances. • Horizontal scalability through 2000, but not beyond. S/W Load Balancer S/W Load Balancer S/W Load Balancer S/W Load Balancer Web Server Web Server Web Server Apache C++ COTS Search OS OS OS UNIX “CGIn” “Listings” “Pages” “Search” RDBMS RDBMS RDBMS RDBMS UNIX UNIX UNIX UNIX 2000 bull.ebay.com bear.ebay.com chard.ebay.com cab/bongo.ebay.com ® ©2008, eBay Inc.
eBay Example #1 Virtualizing the Database Application Servers CATY Attributes Catalogs Rules User Account Feedback Misc API Scratch 1…N DB 1 DB 2 DB 3 • Separate Application notion of a database from physical implementation • Databases may be combined and separated with no code changes • Reduce cost of creating multiple environments (Dev, QA, …) • Application can continue to function without non-critical data (markdown) ® ©2008, eBay Inc.
eBay Example #1 Virtualizing & Scaling the Database November, 1999 November, 1999 ® ©2008, eBay Inc.
eBay Example #1 Virtualizing & Scaling the Database December, 2002 SAN ® ©2008, eBay Inc.
eBay Example #1 Virtualizing & Scaling the Database • Scales Out – 241 million registered users – 112 million plus Items – 7 million plus new items per day – 50 billion SQL transactions per day – 600+ production database instances (inc replicas) – 100+ clusters • Cheaper – Smaller, potentially commodity, servers • Highly Resilient – 2-4 copies of everything – Minimized impact of outage to [relatively] small sub-set of data • Flexible/Agile – Easy to change – database, schemas, partitioning etc. – Minimal impact on architecture or code ® ©2008, eBay Inc.
eBay Example #2 Scaling The Application ® ©2008, eBay Inc.
eBay Example #2 Scaling The Application • Partition code into functional areas – Application is specific to a single area (Buying, Selling etc.) – Domain contains common business logic across applications • Restrict inter-dependencies – Applications depend on Domains, not on other applications – No dependencies among shared domains User Application User Application Selling Application Selling Application Buying Application Buying Application Billing Application Billing Application Search Application Search Application Applications User Domain User Domain Selling Domain Selling Domain Buying Domain Buying Domain Billing Domain Billing Domain Search Domain Search Domain Personalization Domain Personalization Domain User Validation Domain User Validation Domain Shared Billing Domain Shared Billing Domain Shared Shared Buying Domain Shared Buying Domain myEBay Domain myEBay Domain Shared Search Domain Shared Search Domain Domains Core Domain Core Domain API Domain API Domain Lookup Domain Lookup Domain ® ©2008, eBay Inc.
eBay Example #2 Scaling The Application • Segment functions into separate application pools – Minimizes/isolates DB dependencies – Allows for parallel development, deployment and monitoring ViewItem Pool SYI Pool http://cgiX.ebay.com... http://cgiY.ebay.com... Load Load Load Load Balancer Balancer Balancer Balancer Web Web Web Web Web Web Web Web Web Servers Load Load Load Load Balancer Balancer Balancer Balancer AS AS AS AS AS AS AS AS AS AS AS AS App Servers Load Load Balancer Balancer User Acct Caty1 Caty20+ ® ©2008, eBay Inc.
eBay Example #2 Scaling The Application • Everything behaves as loosely coupled services • Minimize inter-dependencies • Infrastructure is like a giant FPGA – Potential to re-program by re-routing traffic • Scales Up And Out (Choice) – Scale out means scaled throughput and resilience – 16000+ concurrent instances – 8000+ servers (mainly blades) • Efficiency – Run traffic from different time zones on the same server but different instances ® ©2008, eBay Inc.
Consequences • Scale Out – Pro – Scale, throughput, resilience, use commodity products – Con – More to manage – complexity, relationships • Virtualization – Pro – Flexibility – Con – more relationships to manage • Commodity – Interchangeable, choice, no lock in, lower unit cost ® ©2008, eBay Inc.
The Big Problem Management complexity scales with this # Relationships # Relationships # Components ® ©2008, eBay Inc.
Understanding Relationships Service A is composed of Persistence Sub-Service B Business Logic Sub-Service C A Presentation Sub-Service D B C D ® ©2008, eBay Inc.
Understanding Relationships Business Logic Sub-Service C is composed of A Load Balancing Service Several Application Instances A B C D App App LBS ® ©2008, eBay Inc.
Understanding Relationships The Application Instances are hosted on Operating System Instances The Load Balancing Service is hosted on A A Load Balancer Operating System B C D App App LBS OS OS LB ® ©2008, eBay Inc.
Understanding Relationships The Operating System Instances are hosted on Servers or Virtual Servers, which are in turn hosted on servers The Load Balancer OS is hosted on A A Physical Load Balancer B C D App App LBS OS OS LB VS Svr Svr LB ® ©2008, eBay Inc.
Interaction/Traffic Relationships Starting To Look Complicated! ® ©2008, eBay Inc.
Relationships Are Everything! • Everything is interconnected • Changing one thing causes ripples • How you connect things together determines business functionality and business value • Agility is the ability to change these relationships dynamically (easier with loosely coupled services) ® ©2008, eBay Inc.
Relationships Are Everything! • Virtualization is about standardizing relationships and interposing/isolating one end from the other • Understanding these relationships allows you to – Tie business processes to the infrastructure they run on – Map value to cost – Understand and manage traffic flow – Understand and manage provisioning etc. • It’s all about managing relationships, not things! ® ©2008, eBay Inc.
Data Center Evolution • Non-Deterministic • Out of band changes • Hosting other people’s code • Emergent Behavior • People and other software using things in unintended ways • SO-Aaaaaaaghhhhh • More relationships ® ©2008, eBay Inc.
Conclusions • NGDC is not just about technology that enables greater scaling, flexibility, resilience etc. • NGDC has to be about changing the nature of the data center and its relationship to the business • The challenge is how to understand and manage relationships, not just things! – Relationships inside the datacenter – Relationships to external services (SaaS et al) – Relationships managed on external infrastructure (Clouds, PaaS) – Dynamically created and destroyed relationships as services come and go! ® ©2008, eBay Inc.
Thank You Paul Strong pstrong@ebay.com Distinguished Research Scientist eBay Research Labs, ® eBay Inc.

More Related Content

Pstrong Cybera 29 Sept 2008

  • 1. eBay – Examples Of NGDC Technologies At Work Paul Strong Distinguished Research Scientist, eBay Research Labs ® Cybera Summit 2008
  • 2. Copyright Notice © 2008 eBay Inc. All rights reserved. • No part of these materials may be reproduced or transmitted in any form, by any means (electronic, photocopying, recording, or otherwise) without the prior permission of eBay Inc. • eBay and the eBay logo are registered trademarks of eBay Inc. • PayPal and the PayPal logo are registered trademarks of PayPal, Inc. • Other trademarks and brands are the property of their respective owners. • Please do not take our picture or record the class/session without asking permission. ® ©2008, eBay Inc.
  • 3. What Is A Next Generation Data Center? • Is it the platform for SOA? • Is it a Grid? • Does it leverage Virtualization technologies? • Is it full of blades? • Is it Greener? • Is it more efficient? • Is it more automated? ® ©2008, eBay Inc.
  • 4. What Is A Next Generation Data Center? • Is it the platform for SOA? • Is it a Grid? • Does it leverage Virtualization technologies? • Is it full of blades? • Is it Greener? • Is it more efficient? • Is it more automated? ® ©2008, eBay Inc.
  • 5. NGDC Is… More Than A Collection Of Technologies • Runs Business Processes driven by SLAs • Is a Value Center, rather than a Cost Center – A new dialog between IT and the Business • Enables – Internal utilities for core business functions/processes – External utilities for business process elements – Businesses based on business process mash-ups – Opportunities for new platforms ® ©2008, eBay Inc.
  • 6. Cloud Computing ® ©2008, eBay Inc.
  • 7. Cloud Computing Is… • A new way about thinking of business… – Business = Codified core differentiating value • Outsourcing everything that is not core – Physical infrastructure → Clouds, Platform as a Service (PaaS) – Logical infrastructure (non-core business software) → Software as a Service (SaaS) • About being flexible – Temporary outsourcing… • Not a new set of technologies – A new context for applying a set of evolving technologies that are now more mature or capable ® ©2008, eBay Inc.
  • 8. Cloud Computing Is About… ® ©2008, eBay Inc.
  • 9. NGDC Is… More Than A Collection Of Technologies • Runs Business Processes driven by SLAs • Is a Value Center, rather than a Cost Center – A new dialog between IT and the Business • Enables – Internal utilities clouds for core business functions/processes – External utilities clouds (PaaS, SaaS) for infrastructure and business process elements – Businesses based on business process mash-ups – Opportunities for new platforms ® ©2008, eBay Inc.
  • 10. NGDC, Friends & Relations U ® ©2008, eBay Inc.
  • 11. Managing NGDCs… Is not about managing servers, operating systems, disks etc. ® ©2008, eBay Inc.
  • 13. Why eBay Is A Useful Example New Challenges Extreme Engineering The Bleeding Edge Everyday use Technology trickle down/transfer ® ©2008, eBay Inc.
  • 14. eBay – The 30 Second Introduction! eBay users trade about $1,989 worth of goods on the site every second On an average day on eBay… • A car sells every 56 seconds • A pair of shoes sell every 7 seconds • A cell phones sells every 7 seconds 1.3m people make all or part of their living selling on * ® ©2008, eBay Inc. *ACNielsen International Research, June 2006
  • 15. eBay’s Drivers • Extreme Scale – 241m Registers Users, 112m+ Items, 7m+ New Items Per Day • Extreme Growth – Near exponential growth in listings for most of history – 13 years • Extreme Agility – Roll code to the site every 2 weeks • Constant, predictable presence – Must be 24x7x365 • Efficiency Failure To Keep Up Is Not An Option! ® ©2008, eBay Inc.
  • 16. eBay – The Simple Diagram ® ©2008, eBay Inc.
  • 17. eBay Example #1 Making The Database Scale • Second Database for failover • CGI pools, Listings, Pages, and Search continued to scale horizontally However … By November 1999, the database servers approached their limits of physical growth. S/W Load Balancer S/W Load Balancer S/W Load Balancer S/W Load Balancer Web Server Web Server Web Server Apache C++ COTS Search OS OS OS UNIX “CGIn” “Listings” “Pages” “Search” RDBMS RDBMS UNIX UNIX 1999 bull.ebay.com bear.ebay.com ® ©2008, eBay Inc.
  • 18. eBay Example #1 Making The Database Scale • Database quot;splitquot; technology. • Logically partition database into separate instances. • Horizontal scalability through 2000, but not beyond. S/W Load Balancer S/W Load Balancer S/W Load Balancer S/W Load Balancer Web Server Web Server Web Server Apache C++ COTS Search OS OS OS UNIX “CGIn” “Listings” “Pages” “Search” RDBMS RDBMS RDBMS RDBMS UNIX UNIX UNIX UNIX 2000 bull.ebay.com bear.ebay.com chard.ebay.com cab/bongo.ebay.com ® ©2008, eBay Inc.
  • 19. eBay Example #1 Virtualizing the Database Application Servers CATY Attributes Catalogs Rules User Account Feedback Misc API Scratch 1…N DB 1 DB 2 DB 3 • Separate Application notion of a database from physical implementation • Databases may be combined and separated with no code changes • Reduce cost of creating multiple environments (Dev, QA, …) • Application can continue to function without non-critical data (markdown) ® ©2008, eBay Inc.
  • 20. eBay Example #1 Virtualizing & Scaling the Database November, 1999 November, 1999 ® ©2008, eBay Inc.
  • 21. eBay Example #1 Virtualizing & Scaling the Database December, 2002 SAN ® ©2008, eBay Inc.
  • 22. eBay Example #1 Virtualizing & Scaling the Database • Scales Out – 241 million registered users – 112 million plus Items – 7 million plus new items per day – 50 billion SQL transactions per day – 600+ production database instances (inc replicas) – 100+ clusters • Cheaper – Smaller, potentially commodity, servers • Highly Resilient – 2-4 copies of everything – Minimized impact of outage to [relatively] small sub-set of data • Flexible/Agile – Easy to change – database, schemas, partitioning etc. – Minimal impact on architecture or code ® ©2008, eBay Inc.
  • 23. eBay Example #2 Scaling The Application ® ©2008, eBay Inc.
  • 24. eBay Example #2 Scaling The Application • Partition code into functional areas – Application is specific to a single area (Buying, Selling etc.) – Domain contains common business logic across applications • Restrict inter-dependencies – Applications depend on Domains, not on other applications – No dependencies among shared domains User Application User Application Selling Application Selling Application Buying Application Buying Application Billing Application Billing Application Search Application Search Application Applications User Domain User Domain Selling Domain Selling Domain Buying Domain Buying Domain Billing Domain Billing Domain Search Domain Search Domain Personalization Domain Personalization Domain User Validation Domain User Validation Domain Shared Billing Domain Shared Billing Domain Shared Shared Buying Domain Shared Buying Domain myEBay Domain myEBay Domain Shared Search Domain Shared Search Domain Domains Core Domain Core Domain API Domain API Domain Lookup Domain Lookup Domain ® ©2008, eBay Inc.
  • 25. eBay Example #2 Scaling The Application • Segment functions into separate application pools – Minimizes/isolates DB dependencies – Allows for parallel development, deployment and monitoring ViewItem Pool SYI Pool http://cgiX.ebay.com... http://cgiY.ebay.com... Load Load Load Load Balancer Balancer Balancer Balancer Web Web Web Web Web Web Web Web Web Servers Load Load Load Load Balancer Balancer Balancer Balancer AS AS AS AS AS AS AS AS AS AS AS AS App Servers Load Load Balancer Balancer User Acct Caty1 Caty20+ ® ©2008, eBay Inc.
  • 26. eBay Example #2 Scaling The Application • Everything behaves as loosely coupled services • Minimize inter-dependencies • Infrastructure is like a giant FPGA – Potential to re-program by re-routing traffic • Scales Up And Out (Choice) – Scale out means scaled throughput and resilience – 16000+ concurrent instances – 8000+ servers (mainly blades) • Efficiency – Run traffic from different time zones on the same server but different instances ® ©2008, eBay Inc.
  • 27. Consequences • Scale Out – Pro – Scale, throughput, resilience, use commodity products – Con – More to manage – complexity, relationships • Virtualization – Pro – Flexibility – Con – more relationships to manage • Commodity – Interchangeable, choice, no lock in, lower unit cost ® ©2008, eBay Inc.
  • 28. The Big Problem Management complexity scales with this # Relationships # Relationships # Components ® ©2008, eBay Inc.
  • 29. Understanding Relationships Service A is composed of Persistence Sub-Service B Business Logic Sub-Service C A Presentation Sub-Service D B C D ® ©2008, eBay Inc.
  • 30. Understanding Relationships Business Logic Sub-Service C is composed of A Load Balancing Service Several Application Instances A B C D App App LBS ® ©2008, eBay Inc.
  • 31. Understanding Relationships The Application Instances are hosted on Operating System Instances The Load Balancing Service is hosted on A A Load Balancer Operating System B C D App App LBS OS OS LB ® ©2008, eBay Inc.
  • 32. Understanding Relationships The Operating System Instances are hosted on Servers or Virtual Servers, which are in turn hosted on servers The Load Balancer OS is hosted on A A Physical Load Balancer B C D App App LBS OS OS LB VS Svr Svr LB ® ©2008, eBay Inc.
  • 33. Interaction/Traffic Relationships Starting To Look Complicated! ® ©2008, eBay Inc.
  • 34. Relationships Are Everything! • Everything is interconnected • Changing one thing causes ripples • How you connect things together determines business functionality and business value • Agility is the ability to change these relationships dynamically (easier with loosely coupled services) ® ©2008, eBay Inc.
  • 35. Relationships Are Everything! • Virtualization is about standardizing relationships and interposing/isolating one end from the other • Understanding these relationships allows you to – Tie business processes to the infrastructure they run on – Map value to cost – Understand and manage traffic flow – Understand and manage provisioning etc. • It’s all about managing relationships, not things! ® ©2008, eBay Inc.
  • 36. Data Center Evolution • Non-Deterministic • Out of band changes • Hosting other people’s code • Emergent Behavior • People and other software using things in unintended ways • SO-Aaaaaaaghhhhh • More relationships ® ©2008, eBay Inc.
  • 37. Conclusions • NGDC is not just about technology that enables greater scaling, flexibility, resilience etc. • NGDC has to be about changing the nature of the data center and its relationship to the business • The challenge is how to understand and manage relationships, not just things! – Relationships inside the datacenter – Relationships to external services (SaaS et al) – Relationships managed on external infrastructure (Clouds, PaaS) – Dynamically created and destroyed relationships as services come and go! ® ©2008, eBay Inc.
  • 38. Thank You Paul Strong pstrong@ebay.com Distinguished Research Scientist eBay Research Labs, ® eBay Inc.