Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
SlideShare a Scribd company logo

4069180 Caching Performance Lessons From Facebook

•
•

This document discusses techniques for improving caching performance at Facebook. It begins by explaining the benefits of caching for large sites and applications. It then describes the various caching layers Facebook uses, including globals caching, APC caching, memcached, and browser caching. It provides examples of how Facebook leverages these caches, such as caching parsed user agent strings, opcode caching with APC, and caching profile data in memcached. It also discusses challenges like cache invalidation between data centers and techniques for addressing them like caching proxies.

1 of 34
Download to read offline
Caching & Performance: Lessons from Facebook Lucas Nealan OSCon, Portland OR July 23, 2008, 14:35 – 15:20
Facebook Social Utility Sharing and communicating efficiently 4th most trafficked site in the World * * ComScore 2008
Facebook Stats Over 90 million active users ~ 50 pages per user daily Over 24,000 platform applications
Facebook Architecture !5678 -**. /012", (") *+", 34%#%+ !"#$%$&"'
Complexity Connecting to all Database is impossible Large codebase Scaling affects resources in many ways ‣ Memory consumption ‣ Socket connection limits Cache retrieval is ~ 10% cpu-user of most pages
What are the Benefits of Caching?
Caching Layers $GLOBALS APC Memcached Database Browser Cache Third Party CDN
Globals Cache function cache_get($id, $key, $apc=false) { if (isset($GLOBALS['CACHE']["$key:$id"])) { $cache = $GLOBALS['CACHE']["$key:$id"])); $hit = 1; } elseif ($apc && (($cache = apc_fetch("$key:$id")) !== false) { $hit = 1; } else { ... // fetch from memcached if ($apc) apc_store("$key:$id", $cache); } if ($hit) $GLOBALS['CACHE']["$key:$id"] = $cache; return $hit ? $cache : NULL; }
Globals Cache Avoids unnecessary APC and Memcached requests Automatic via abstraction But still has function call cost overhead foreach($ids as $id) { if (isset($GLOBALS['CACHE']["profile:$id")) { $profile = $GLOBALS['CACHE']["profile:$id"]; } else { $profile = cache_get($id, 'profile'); } }
APC Opcode caching ‣ Hundreds of included libraries ‣ Thousands of functions Variable caching ‣ Hundreds of MB’s of data
APC User Cache Non-user specific data ‣ Network/School information ‣ Database information ‣ Useragent strings ‣ Hot application data ‣ Site variables ‣ Languange Strings
Friends Normal 4050ms APC 135ms apc.stat=0 128ms
APC Opcode Priming $path = '/path/to/source'; $exclude = $path.'/www/admin/'; $expr = '.*.php'; $files = split(' ', exec("find -L $path -regex '$expr' | grep -v '$exclude' | xargs")); // prime php files foreach($files as $file) { apc_compile_file($file); }
APC+SVN Client Cache Busting function get_static_suffix($file) { global $ROOT; if ($version = cache_get($file, ‘sv’, 1) === null) { $version = trim(shell_exec(“svn info $ROOT/$file | grep ‘Changed Rev’ | cut -c 19-”)); apc_store(“sv:$file”, $version); } return '?'.$version; }
APC + Useragent Strings Useragent string parsing is inefficient in PHP Cache parsed useragents in APC for the first 10 minutes Hit rate of over 50% Pear implementation available: ‣ PEAR::Net_Useragent_Detect_APC
Site Variables Enable/Disable site features across all servers Configure memcached cluster IP’s Configure product features Version memcached keys for invalidation
Site Variables 9+;$6<-87-8 !"# -.(&91<67686 6'"5$'7686 -./01$23 -.9&98$7#:# -./01$23 -./.$'-"+, -./.$'-"+,4#"5$ -./.$'-"+, -.<6;&#$7686 =$59&98$; *+;$ B<-8 !$'.$' =2!>? -.(&91<67686 !"#$.&' @A !"#$ %&'"&()$ *+,-+)$
Memcached Distributed object cache Facebook currently utilizes > 400 memcached hosts With > 5TB in memory cache Facebook contributions: ‣ UDP Support ‣ Performance Enhancements Many choices in opensource clients
What to cache? User Specific Data ‣ Long profile ‣ Short profile ‣ Friends ‣ Applications Key versioning ‣ sp:6:10030226
Cache Retreival Create Wrapper functions: ‣ cache_get($id, <key>, <miss>, $apc, $timeout); ‣ cache_get_multi($ids, <key>, <miss>, $apc); Cache key callback function: function profile_key($id) { global $VERSION_SP; // primed site variable return “sp:$VERSION_SP:$id”; }
Cache Multiget Sort keys into buckets of servers foreach($keys as $key => $kdata) { $host = get_host($key); // hash For each server $keys_hosts[$host][$key] = $kdata; ‣ obtain connection } ‣ send requests function get_host($key) { global $servers; For each server ‣ read responses $prefix = $key[0].$key[1].$key[2]; $prefix = isset($servers[$prefix) ? ‣ deserialize non-scalar types $prefix : ‘wildcard’; $hash = crc32($key); $host = $servers[$hash % count($servers[$prefix])]; }
Cache Multiget Sort keys into buckets of servers if (isset($cache_sock[$host])) { return $cache_sock[$host]; For each server } ‣ obtain connection list($ip, $port) = explode(‘:’, $host); ‣ send requests while($try < 5 && !$sock) { For each server $sock = pfsockopen($ip, $port ...); ‣ read responses $try ++; } ‣ deserialize non-scalar types stream_set_write_buffer($sock, 0); $cmd = “get $keysrn”; fwrite($sock, $cmd);
Profile Multigets Profile info Profile installed platform applications Viewer installed applications Platform application data List of friends Privacy data
Cache Dispatching cache_get_multi($ids, <key>, <miss>, $apc, $pending=false, &$pending_arr); cache_dispatch(); Combine requests for data from the same memcache server ‣ Up to 10% performance improvement Execute code while the kernel buffers the memcache response
Profile Multigets <?php <?php include_once(...); include_once(...); •get_profile(true); parse_arguments(); •get_photos(true); check_permissions(); •get_applications(true) check_friends(); •get_friends(true) check_friend_status(); •get_networks(true) • get_profile(); cache_dispatch(); render_basic_information(); get_friend_details(); parse_arguments(); render_minifeed(); check_permissions(); render_wall(); check_friends(); • get_photos(); check_friend_status(); count_photos(); • get_profile(); • get_applications(); render_basic_information(); render_menu_actions(); get_friend_details(); • get_friends(); render_minifeed(); render_friends(); render_wall(); • get_networks(); get_photos(); render_networks(); count_photos(); render_applications(); get_applications(); render_menu_actions(); get_friends(); render_friends(); get_networks(); render_networks(); render_applications();
Lets make it even faster Memcached PHP extension ‣ Reduced PHP function calling overhead ‣ Socket blocking in C instead of userspace PHP ‣ UDP support
Friends Again Memcached extension runs ~ 10% faster realtime than in PHP userspace Userspace 131ms Extension 115ms Extension w/ 122ms UDP
Serialization Facebook internal serialization ‣ Profiles store 38% less memory in memcache ‣ Improves network throughput and realtime I/O Compression ‣ Investigating LZO compression for even more space savings
UDP Memcached TCP Limitations ‣ Maximum simultaneous connect count of ~ 250k ‣ Impedes scalability of memcached clusters Requires a very stable network environment Occasional misses are acceptible Reduces kernel buffer memory usage on clients and servers Supported in Facebook Memcache Extension coming soon
A Dirty Problem Wrapper functions ‣ cache_dirty($id, $key); Actively dirty cache entries as users change data Dirty entries between multiple hosting facilities via proxy
Memcached Proxy 0"1 2"34"3 56%$&" !.2=: !"#$%$&"' () 7389. :8; ,%-"<$. *%$+,+-. (/
Latency Proxy deletes only work with low latency between facilities When facilities are further apart deletes need to be smarter 5"6 /"78"7 92%$&" !./0+ !"#$%$&"' () +%,"-$. !./0+ 1"234$% !"#$%$&"' (*
Presention online http://sizzo.org/talks/ lucas@facebook.com
4069180 Caching Performance Lessons From Facebook

More Related Content

4069180 Caching Performance Lessons From Facebook

  • 1. Caching & Performance: Lessons from Facebook Lucas Nealan OSCon, Portland OR July 23, 2008, 14:35 – 15:20
  • 2. Facebook Social Utility Sharing and communicating efficiently 4th most trafficked site in the World * * ComScore 2008
  • 3. Facebook Stats Over 90 million active users ~ 50 pages per user daily Over 24,000 platform applications
  • 4. Facebook Architecture !5678 -**. /012", (") *+", 34%#%+ !"#$%$&"'
  • 5. Complexity Connecting to all Database is impossible Large codebase Scaling affects resources in many ways ‣ Memory consumption ‣ Socket connection limits Cache retrieval is ~ 10% cpu-user of most pages
  • 6. What are the Benefits of Caching?
  • 7. Caching Layers $GLOBALS APC Memcached Database Browser Cache Third Party CDN
  • 8. Globals Cache function cache_get($id, $key, $apc=false) { if (isset($GLOBALS['CACHE']["$key:$id"])) { $cache = $GLOBALS['CACHE']["$key:$id"])); $hit = 1; } elseif ($apc && (($cache = apc_fetch("$key:$id")) !== false) { $hit = 1; } else { ... // fetch from memcached if ($apc) apc_store("$key:$id", $cache); } if ($hit) $GLOBALS['CACHE']["$key:$id"] = $cache; return $hit ? $cache : NULL; }
  • 9. Globals Cache Avoids unnecessary APC and Memcached requests Automatic via abstraction But still has function call cost overhead foreach($ids as $id) { if (isset($GLOBALS['CACHE']["profile:$id")) { $profile = $GLOBALS['CACHE']["profile:$id"]; } else { $profile = cache_get($id, 'profile'); } }
  • 10. APC Opcode caching ‣ Hundreds of included libraries ‣ Thousands of functions Variable caching ‣ Hundreds of MB’s of data
  • 11. APC User Cache Non-user specific data ‣ Network/School information ‣ Database information ‣ Useragent strings ‣ Hot application data ‣ Site variables ‣ Languange Strings
  • 12. Friends Normal 4050ms APC 135ms apc.stat=0 128ms
  • 13. APC Opcode Priming $path = '/path/to/source'; $exclude = $path.'/www/admin/'; $expr = '.*.php'; $files = split(' ', exec("find -L $path -regex '$expr' | grep -v '$exclude' | xargs")); // prime php files foreach($files as $file) { apc_compile_file($file); }
  • 14. APC+SVN Client Cache Busting function get_static_suffix($file) { global $ROOT; if ($version = cache_get($file, ‘sv’, 1) === null) { $version = trim(shell_exec(“svn info $ROOT/$file | grep ‘Changed Rev’ | cut -c 19-”)); apc_store(“sv:$file”, $version); } return '?'.$version; }
  • 15. APC + Useragent Strings Useragent string parsing is inefficient in PHP Cache parsed useragents in APC for the first 10 minutes Hit rate of over 50% Pear implementation available: ‣ PEAR::Net_Useragent_Detect_APC
  • 16. Site Variables Enable/Disable site features across all servers Configure memcached cluster IP’s Configure product features Version memcached keys for invalidation
  • 17. Site Variables 9+;$6<-87-8 !"# -.(&91<67686 6'"5$'7686 -./01$23 -.9&98$7#:# -./01$23 -./.$'-"+, -./.$'-"+,4#"5$ -./.$'-"+, -.<6;&#$7686 =$59&98$; *+;$ B<-8 !$'.$' =2!>? -.(&91<67686 !"#$.&' @A !"#$ %&'"&()$ *+,-+)$
  • 18. Memcached Distributed object cache Facebook currently utilizes > 400 memcached hosts With > 5TB in memory cache Facebook contributions: ‣ UDP Support ‣ Performance Enhancements Many choices in opensource clients
  • 19. What to cache? User Specific Data ‣ Long profile ‣ Short profile ‣ Friends ‣ Applications Key versioning ‣ sp:6:10030226
  • 20. Cache Retreival Create Wrapper functions: ‣ cache_get($id, <key>, <miss>, $apc, $timeout); ‣ cache_get_multi($ids, <key>, <miss>, $apc); Cache key callback function: function profile_key($id) { global $VERSION_SP; // primed site variable return “sp:$VERSION_SP:$id”; }
  • 21. Cache Multiget Sort keys into buckets of servers foreach($keys as $key => $kdata) { $host = get_host($key); // hash For each server $keys_hosts[$host][$key] = $kdata; ‣ obtain connection } ‣ send requests function get_host($key) { global $servers; For each server ‣ read responses $prefix = $key[0].$key[1].$key[2]; $prefix = isset($servers[$prefix) ? ‣ deserialize non-scalar types $prefix : ‘wildcard’; $hash = crc32($key); $host = $servers[$hash % count($servers[$prefix])]; }
  • 22. Cache Multiget Sort keys into buckets of servers if (isset($cache_sock[$host])) { return $cache_sock[$host]; For each server } ‣ obtain connection list($ip, $port) = explode(‘:’, $host); ‣ send requests while($try < 5 && !$sock) { For each server $sock = pfsockopen($ip, $port ...); ‣ read responses $try ++; } ‣ deserialize non-scalar types stream_set_write_buffer($sock, 0); $cmd = “get $keysrn”; fwrite($sock, $cmd);
  • 23. Profile Multigets Profile info Profile installed platform applications Viewer installed applications Platform application data List of friends Privacy data
  • 24. Cache Dispatching cache_get_multi($ids, <key>, <miss>, $apc, $pending=false, &$pending_arr); cache_dispatch(); Combine requests for data from the same memcache server ‣ Up to 10% performance improvement Execute code while the kernel buffers the memcache response
  • 25. Profile Multigets <?php <?php include_once(...); include_once(...); •get_profile(true); parse_arguments(); •get_photos(true); check_permissions(); •get_applications(true) check_friends(); •get_friends(true) check_friend_status(); •get_networks(true) • get_profile(); cache_dispatch(); render_basic_information(); get_friend_details(); parse_arguments(); render_minifeed(); check_permissions(); render_wall(); check_friends(); • get_photos(); check_friend_status(); count_photos(); • get_profile(); • get_applications(); render_basic_information(); render_menu_actions(); get_friend_details(); • get_friends(); render_minifeed(); render_friends(); render_wall(); • get_networks(); get_photos(); render_networks(); count_photos(); render_applications(); get_applications(); render_menu_actions(); get_friends(); render_friends(); get_networks(); render_networks(); render_applications();
  • 26. Lets make it even faster Memcached PHP extension ‣ Reduced PHP function calling overhead ‣ Socket blocking in C instead of userspace PHP ‣ UDP support
  • 27. Friends Again Memcached extension runs ~ 10% faster realtime than in PHP userspace Userspace 131ms Extension 115ms Extension w/ 122ms UDP
  • 28. Serialization Facebook internal serialization ‣ Profiles store 38% less memory in memcache ‣ Improves network throughput and realtime I/O Compression ‣ Investigating LZO compression for even more space savings
  • 29. UDP Memcached TCP Limitations ‣ Maximum simultaneous connect count of ~ 250k ‣ Impedes scalability of memcached clusters Requires a very stable network environment Occasional misses are acceptible Reduces kernel buffer memory usage on clients and servers Supported in Facebook Memcache Extension coming soon
  • 30. A Dirty Problem Wrapper functions ‣ cache_dirty($id, $key); Actively dirty cache entries as users change data Dirty entries between multiple hosting facilities via proxy
  • 31. Memcached Proxy 0"1 2"34"3 56%$&" !.2=: !"#$%$&"' () 7389. :8; ,%-"<$. *%$+,+-. (/
  • 32. Latency Proxy deletes only work with low latency between facilities When facilities are further apart deletes need to be smarter 5"6 /"78"7 92%$&" !./0+ !"#$%$&"' () +%,"-$. !./0+ 1"234$% !"#$%$&"' (*