Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
SlideShare a Scribd company logo

Netflix CDN and Open Source

•
•
G
Gleb Smirnoff

Netflix operates a large content delivery network (CDN) to stream video to over 50 million subscribers globally. To reduce costs and improve performance, Netflix built its own CDN called OpenConnect, using open source software like FreeBSD and Nginx. Netflix has significantly improved the performance of OpenConnect over time by contributions to open source projects, such as reducing locking contention and improving networking, storage, and virtual memory subsystems. Netflix aims to continue optimizing OpenConnect and working with open source communities to achieve its goal of over 80Gbps performance per appliance.

1 of 45
Downloaded 407 times
Introduction Net ix CDN and Open Source Gleb Smirno glebius@nginx.com Nginx.conf 2014 San Francisco October 21, 2014 Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 1 / 1
Introduction to Net ix CDN Net ix in numbers I 50+ millions subscribers in 40+ countries I Over 1 Petabyte of data Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 2 / 1
Introduction to Net ix CDN Net ix trac 2014 1H Sandvine North Amertica Trac Report Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 3 / 1
Net ix CDN Net ix Streaming network Components of Net ix Streaming I Amazon Web Services I Website, Business Logic, Customer Authentication I Data Sciences, Encoding/Encryption I Command and Control I Content Streaming I Big 3 CDN's (Akamai, Limelight, Level3) I Own CDN Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 4 / 1
Net ix CDN Running own CDN Reasons for running own CDN I Grow faster I Reduce costs 1https://www.net ix.com/openconnect Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 5 / 1
Net ix CDN Running own CDN Reasons for running own CDN I Grow faster I Reduce costs I Control the server side of HTTP connection I Build a CDN specialized in Net ix content delivery I Put the content closer to a client 1https://www.net ix.com/openconnect Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 5 / 1
Net ix CDN Running own CDN Reasons for running own CDN I Grow faster I Reduce costs I Control the server side of HTTP connection I Build a CDN specialized in Net ix content delivery I Put the content closer to a client Solution: Open Connect1 1https://www.net ix.com/openconnect Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 5 / 1
OpenConnect Introduction to OpenConnect The OpenConnect Initiative Amazon Cloud The Internet ISPs video encoding account authorization streaming control Origin Server ISP Cache IX Cache Client Client Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 6 / 1
OpenConnect Introduction to OpenConnect OpenConnect Appliance Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 7 / 1
OpenConnect OpenConnect hardware OpenConnect Appliance The big cache I 4U chassis I Storage I 36 HDDs, 4 Terabyte each I 6 SSDs, 0.5 Terabyte each I Two 10 Gbit/s Chelsio NICs I 8-core CPU Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 8 / 1
OpenConnect OpenConnect hardware OpenConnect Appliance The fast cache I 1U chassis I Storage I 14 SSDs, 1 Terabyte each I 40 Gbit/s Chelsio NICs I 8-core CPU Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 8 / 1
OpenConnect OpenConnect software OpenConnect Appliance software I FreeBSD operating system I nanobsd(8) based
rmware I some enhancements I NGINX web server I custom modules I BIRD routing daemon Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 9 / 1
OpenConnect OpenConnect software Software choice I Open source I BSD license Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 10 / 1
OpenConnect OpenConnect software Software choice I Open source I BSD license I FreeBSD I Known to be fast and stable I Strong developer community Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 10 / 1
OpenConnect OpenConnect software Software choice I Open source I BSD license I FreeBSD I Known to be fast and stable I Strong developer community I NGINX I Known to be fast and stable I Commercial support from Nginx, Inc. I Flexible framework for custom modules Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 10 / 1
OpenConnect OpenConnect software Software choice I Open source I BSD license I FreeBSD I Known to be fast and stable I Strong developer community I NGINX I Known to be fast and stable I Commercial support from Nginx, Inc. I Flexible framework for custom modules I FreeBSD + NGINX I kqueue(2) event driven model I send
le(2) + aio(4) mode Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 10 / 1
OpenConnect OpenConnect operation OpenConnect operation: serve
ll Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 11 / 1
OpenConnect OpenConnect operation OpenConnect: streaming numbers I 400,000 stream
les per appliance I 5000-30,000 client streams per appliance I 300-1000 clients per disk I Clients request 80Kb-300Kb ranges Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 12 / 1
Net ix open source Dealing with open source Dealing with open source The traditional way: I Take a stable, well tested version, fork it I Develop your product on top of it Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 13 / 1
Net ix open source Dealing with open source Dealing with open source The traditional way: I Take a stable, well tested version, fork it I Develop your product on top of it Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 13 / 1
Net ix open source Dealing with open source Dealing with open source The Net ix rules: I Pull the bleeding-edge version of software I Push your changes upstream aggressively Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 14 / 1
Net ix open source Dierent myths on dierent versions Myths about development version Myth #1: Development version is full of bugs Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 15 / 1
Net ix open source Dierent myths on dierent versions Myths about development version Myth #1: Development version is full of bugs Truth: most bugs are discovered in stable versions Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 15 / 1
Net ix open source Dierent myths on dierent versions Myths about stable version Myth #2: We will wait for a stable version, and someone else will
x bugs 2 2http://en.wikipedia.org/wiki/Free rider problem Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 16 / 1
Net ix open source Dierent myths on dierent versions Myths about stable version Myth #2: We will wait for a stable version, and someone else will
x bugs 2 Truth: No one will discover your bugs 2http://en.wikipedia.org/wiki/Free rider problem Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 16 / 1
Net ix open source Dierent myths on dierent versions Discovering bugs in dierent versions development stable code is hot code is unmaintained no API/ABI constraints API/ABI must be preserved Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 17 / 1
Net ix open source Dierent myths on dierent versions More myths Myth #3: Not following development version saves us a lot of time Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 18 / 1
Net ix open source Dierent myths on dierent versions More myths Myth #3: Not following development version saves us a lot of time Truth: some day you will need to go forward Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 18 / 1
Net ix open source Opening sources Myths on open source Myth #4: Sharing code discloses know-how's Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 19 / 1
Net ix open source Opening sources Myths on open source Myth #4: Sharing code discloses know-how's Truth: know-how's reside in a tiny percent of code, or even outside Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 19 / 1
Net ix open source Opening sources Noble or sel
sh? I We want to in uence the direction of open source development I We want to outsource maintainance of our code to community I We want more eyes to examine our code I We want more people to discover bugs in it Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 20 / 1
Net ix open source Opening sources Noble or sel
sh? I We want to in uence the direction of open source development I We want to outsource maintainance of our code to community I We want more eyes to examine our code I We want more people to discover bugs in it I And, of course, we want to be considered noble givers of code to community , Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 20 / 1
Net ix open source Net ix's commitment to open source OpenConnect performance I OpenConnect started in 2011 10 Gbps per appliance Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 21 / 1
Net ix open source Net ix's commitment to open source OpenConnect performance I OpenConnect started in 2011 10 Gbps per appliance I Now, in 2014, we achieve 30 Gbps per appliance Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 21 / 1
Net ix open source Net ix's commitment to open source OpenConnect performance I OpenConnect started in 2011 10 Gbps per appliance I Now, in 2014, we achieve 30 Gbps per appliance I Next goal is 80 Gbps per appliance Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 21 / 1
Net ix open source Net ix's commitment to open source Areas of focus I network stack: sockets, TCP, drivers I storage: drivers, UFS I VM subsystem: caching Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 22 / 1
Net ix open source Net ix's commitment to open source Performance improvements I Reducing lock/cache line contention in I link aggregation driver I kernel socket buers I send

More Related Content

Netflix CDN and Open Source

  • 1. Introduction Net ix CDN and Open Source Gleb Smirno glebius@nginx.com Nginx.conf 2014 San Francisco October 21, 2014 Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 1 / 1
  • 2. Introduction to Net ix CDN Net ix in numbers I 50+ millions subscribers in 40+ countries I Over 1 Petabyte of data Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 2 / 1
  • 3. Introduction to Net ix CDN Net ix trac 2014 1H Sandvine North Amertica Trac Report Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 3 / 1
  • 4. Net ix CDN Net ix Streaming network Components of Net ix Streaming I Amazon Web Services I Website, Business Logic, Customer Authentication I Data Sciences, Encoding/Encryption I Command and Control I Content Streaming I Big 3 CDN's (Akamai, Limelight, Level3) I Own CDN Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 4 / 1
  • 5. Net ix CDN Running own CDN Reasons for running own CDN I Grow faster I Reduce costs 1https://www.net ix.com/openconnect Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 5 / 1
  • 6. Net ix CDN Running own CDN Reasons for running own CDN I Grow faster I Reduce costs I Control the server side of HTTP connection I Build a CDN specialized in Net ix content delivery I Put the content closer to a client 1https://www.net ix.com/openconnect Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 5 / 1
  • 7. Net ix CDN Running own CDN Reasons for running own CDN I Grow faster I Reduce costs I Control the server side of HTTP connection I Build a CDN specialized in Net ix content delivery I Put the content closer to a client Solution: Open Connect1 1https://www.net ix.com/openconnect Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 5 / 1
  • 8. OpenConnect Introduction to OpenConnect The OpenConnect Initiative Amazon Cloud The Internet ISPs video encoding account authorization streaming control Origin Server ISP Cache IX Cache Client Client Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 6 / 1
  • 9. OpenConnect Introduction to OpenConnect OpenConnect Appliance Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 7 / 1
  • 10. OpenConnect OpenConnect hardware OpenConnect Appliance The big cache I 4U chassis I Storage I 36 HDDs, 4 Terabyte each I 6 SSDs, 0.5 Terabyte each I Two 10 Gbit/s Chelsio NICs I 8-core CPU Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 8 / 1
  • 11. OpenConnect OpenConnect hardware OpenConnect Appliance The fast cache I 1U chassis I Storage I 14 SSDs, 1 Terabyte each I 40 Gbit/s Chelsio NICs I 8-core CPU Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 8 / 1
  • 12. OpenConnect OpenConnect software OpenConnect Appliance software I FreeBSD operating system I nanobsd(8) based
  • 13. rmware I some enhancements I NGINX web server I custom modules I BIRD routing daemon Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 9 / 1
  • 14. OpenConnect OpenConnect software Software choice I Open source I BSD license Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 10 / 1
  • 15. OpenConnect OpenConnect software Software choice I Open source I BSD license I FreeBSD I Known to be fast and stable I Strong developer community Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 10 / 1
  • 16. OpenConnect OpenConnect software Software choice I Open source I BSD license I FreeBSD I Known to be fast and stable I Strong developer community I NGINX I Known to be fast and stable I Commercial support from Nginx, Inc. I Flexible framework for custom modules Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 10 / 1
  • 17. OpenConnect OpenConnect software Software choice I Open source I BSD license I FreeBSD I Known to be fast and stable I Strong developer community I NGINX I Known to be fast and stable I Commercial support from Nginx, Inc. I Flexible framework for custom modules I FreeBSD + NGINX I kqueue(2) event driven model I send
  • 18. le(2) + aio(4) mode Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 10 / 1
  • 19. OpenConnect OpenConnect operation OpenConnect operation: serve
  • 20. ll Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 11 / 1
  • 21. OpenConnect OpenConnect operation OpenConnect: streaming numbers I 400,000 stream
  • 22. les per appliance I 5000-30,000 client streams per appliance I 300-1000 clients per disk I Clients request 80Kb-300Kb ranges Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 12 / 1
  • 23. Net ix open source Dealing with open source Dealing with open source The traditional way: I Take a stable, well tested version, fork it I Develop your product on top of it Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 13 / 1
  • 24. Net ix open source Dealing with open source Dealing with open source The traditional way: I Take a stable, well tested version, fork it I Develop your product on top of it Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 13 / 1
  • 25. Net ix open source Dealing with open source Dealing with open source The Net ix rules: I Pull the bleeding-edge version of software I Push your changes upstream aggressively Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 14 / 1
  • 26. Net ix open source Dierent myths on dierent versions Myths about development version Myth #1: Development version is full of bugs Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 15 / 1
  • 27. Net ix open source Dierent myths on dierent versions Myths about development version Myth #1: Development version is full of bugs Truth: most bugs are discovered in stable versions Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 15 / 1
  • 28. Net ix open source Dierent myths on dierent versions Myths about stable version Myth #2: We will wait for a stable version, and someone else will
  • 29. x bugs 2 2http://en.wikipedia.org/wiki/Free rider problem Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 16 / 1
  • 30. Net ix open source Dierent myths on dierent versions Myths about stable version Myth #2: We will wait for a stable version, and someone else will
  • 31. x bugs 2 Truth: No one will discover your bugs 2http://en.wikipedia.org/wiki/Free rider problem Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 16 / 1
  • 32. Net ix open source Dierent myths on dierent versions Discovering bugs in dierent versions development stable code is hot code is unmaintained no API/ABI constraints API/ABI must be preserved Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 17 / 1
  • 33. Net ix open source Dierent myths on dierent versions More myths Myth #3: Not following development version saves us a lot of time Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 18 / 1
  • 34. Net ix open source Dierent myths on dierent versions More myths Myth #3: Not following development version saves us a lot of time Truth: some day you will need to go forward Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 18 / 1
  • 35. Net ix open source Opening sources Myths on open source Myth #4: Sharing code discloses know-how's Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 19 / 1
  • 36. Net ix open source Opening sources Myths on open source Myth #4: Sharing code discloses know-how's Truth: know-how's reside in a tiny percent of code, or even outside Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 19 / 1
  • 37. Net ix open source Opening sources Noble or sel
  • 38. sh? I We want to in uence the direction of open source development I We want to outsource maintainance of our code to community I We want more eyes to examine our code I We want more people to discover bugs in it Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 20 / 1
  • 39. Net ix open source Opening sources Noble or sel
  • 40. sh? I We want to in uence the direction of open source development I We want to outsource maintainance of our code to community I We want more eyes to examine our code I We want more people to discover bugs in it I And, of course, we want to be considered noble givers of code to community , Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 20 / 1
  • 41. Net ix open source Net ix's commitment to open source OpenConnect performance I OpenConnect started in 2011 10 Gbps per appliance Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 21 / 1
  • 42. Net ix open source Net ix's commitment to open source OpenConnect performance I OpenConnect started in 2011 10 Gbps per appliance I Now, in 2014, we achieve 30 Gbps per appliance Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 21 / 1
  • 43. Net ix open source Net ix's commitment to open source OpenConnect performance I OpenConnect started in 2011 10 Gbps per appliance I Now, in 2014, we achieve 30 Gbps per appliance I Next goal is 80 Gbps per appliance Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 21 / 1
  • 44. Net ix open source Net ix's commitment to open source Areas of focus I network stack: sockets, TCP, drivers I storage: drivers, UFS I VM subsystem: caching Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 22 / 1
  • 45. Net ix open source Net ix's commitment to open source Performance improvements I Reducing lock/cache line contention in I link aggregation driver I kernel socket buers I send
  • 46. le(2) kernel memory buers Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 23 / 1
  • 47. Net ix open source Net ix's commitment to open source Performance improvements I Reducing lock/cache line contention in I link aggregation driver I kernel socket buers I send
  • 48. le(2) kernel memory buers I Reducing complexity, structuring code in I kernel owtable I NGINX send
  • 49. le(2) code Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 23 / 1
  • 50. Net ix open source Net ix's commitment to open source Performance improvements I Reducing lock/cache line contention in I link aggregation driver I kernel socket buers I send
  • 51. le(2) kernel memory buers I Reducing complexity, structuring code in I kernel owtable I NGINX send
  • 52. le(2) code I Introducing multithreading in I UFS softupdates Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 23 / 1
  • 53. Net ix open source Net ix's commitment to open source Various improvements bug
  • 54. xes I NGINX core and modules I IPv6 network stack I UFS journaling Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 24 / 1
  • 55. Net ix open source Net ix's commitment to open source Completely new features I Per-CPU statistical counters: I Precise: do not lose updates I Fast: do not use any locking, neither critical sections Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 25 / 1
  • 56. Net ix open source Net ix's commitment to open source Completely new features I Per-CPU statistical counters: I Precise: do not lose updates I Fast: do not use any locking, neither critical sections I Completely new send
  • 57. le(2) implementation: I Doesn't block on disk I/O! I Allows to specify readahead I Allows to deny VM caching Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 25 / 1
  • 58. Net ix open source Net ix's commitment to open source Working with community I unmapped I/O I VM radix Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 26 / 1
  • 59. Net ix open source Net ix's commitment to open source Future and work in progress I cc net ix: new TCP congestion control algorithm I hardware assisted TCP pacing I kernel-side TLS ooad I SSD I/O performance improvements I multithreading pagedaemon I NUMA support Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 27 / 1
  • 60. Net ix open source Net ix's commitment to open source Questions? Gleb Smirno glebius@nginx.com Net ix CDN and Open Source October 21, 2014 28 / 1