Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
SlideShare a Scribd company logo

Meetup on Apache Zookeeper

•Download as PPTX, PDF•
•
Anshul Patel
Anshul Patel

Zookeeper is a distributed coordination service that provides naming, configuration, synchronization, and group services. It allows distributed processes to coordinate with each other through a shared hierarchical namespace of data registers called znodes. Zookeeper follows a leader-elected consensus protocol to guarantee atomic broadcast of state updates from the leader to followers. It uses a hierarchical namespace of znodes similar to a file system to store configuration data and other application-defined metadata. Zookeeper provides services like leader election, group membership, synchronization, and configuration management that are essential for distributed systems.

1 of 30
Apache Zookeeper Anshul Patel
What is distributed system ? • A distributed systems provides single coherent view of collection of independent computing machines. • Multiple software components on multiple computers, which run as a single system. • It can be locally or geographically connected.
Characteristics of Distributed System • Concurrency • Fault Tolerance • Scalability • Transparency • Heterogeneity
Delusion of Distributed System • Reliable Network • Zero Latency • Infinite Bandwidth • Secure Network • Topology Doesn’t change • Homogeneous Network • There is one administrator
Introducing Zookeeper • Zookeeper provides coordination between distributed process via shared hierarchical namespace.
Why Zookeeper ? • Naming Service • Locking • Synchronization • Configuration Management • Leader Election
Zookeeper Architecture
Zookeeper Architecture • Zookeeper follows client server architecture • Collection of Zookeeper servers is called an ensemble • At given time, a Zookeeper client is connected to one Zookeeper server • Client read requests are catered by connected Zookeeper server • Client write requests are forwarded to Zookeeper leader. • Write is considered successful, if quorum is achieved.
Zookeeper Architecture • Client updates are sequentially consistent • Updates are atomic • Updates are reliable • Timeliness
ZAB Protocol • Zookeeper Atomic Broadcast • Propagates state changes from leader to followers • Guarantees the state change
Znodes • Organized similar to standard Unix and Linux file system • Similar to files and directories • Both directory and leaf znode holds data • Stores data <= 1MB
Znode - Hierarchical Namespace
Time in Zookeeper • Zxid : Every change to ZK state receives a stamp in form of zxid • Version Number : Change to Znode will cause change in Version • Ticks : Ticks define timing of events • Real Time : To put timestamp of stat structure and znode modification
Znode Stat Structure • Czxid : Change that caused Znode to be created • Mzxid : Change that last modified Znode • Pzxid : Change that last modified children of the Znode • Ctime : Millisecond from epoch, when this Znode was created • Mtime : Millisecond from epoch, when this Znode was last modified • Version : Number of changes to data of Znode • Cversion : Number of changes to children of Znode • Aversion : Number of changes to ACL of Znode • EphemeralOwner : SessionId of owner of this Znode • DataLength : Length of data field • Numchildren : Number of children
Zookeeper Operations create Creates Znode delete Deletes Znode exists Does Znode exist? getACL Get ACL permissions setACL Set ACL permissions
Zookeeper Operations getChildren Gets the childrens of Znode getData Gets Data of Znode setData Sets Data of Znode Sync Syncs the client View
Types of Znodes • Ephemeral Znode : Deleted with termination of client session, also can’t have any child znode. • Persistent Znode : Can have child znode, It is explicitly required to delete it. • Sequential Znode : Append a monotonicly increasing counter to the end of path. It is unique to parent znode.
Znode Watchers • One time triggers • Sent to client which sets the watch • Maintained locally at zookeeper • Client won’t receive watches after getting disconnected • Two lists of watches – Data watches : create(),setData(), delete() – Child watches : create(), delete()
Zookeeper ACL • ACL Permission – Create – Read – Write – Delete – Admin
Zookeeper ACL • ACL Scheme – World – Auth – Digest – Host – IP
Zookeeper Recipes/Higher Order Functions/Apache Curator • Naming Service • Service Discovery • Configuration Management • Locks • 2-phase commit • Leader Election • Barriers • Queues
Netflix/Soabase Exhibitor • Zookeeper server is fail fast • Zookeeper is self-healing, (i.e. crashed/failed server if brought online, will automatically join the cluster) • Instance Monitoring • Log Cleanup • Backup/Restore • Cluster-wide configuration • Rolling Ensemble Change • Rich REST API
Backup/Exhibitor • Online Backup - Use Netflix Exhibitor • Offline Backup - Bring down the old ensemble cluster and copy the zookeeper data directory to the new ensemble cluster
Zookeeper Monitoring • 4 Letter Words • JMX
Zookeeper Monitoring - 4 Letter Words conf srst cons srvr dump stat envi dirs (3.5.1) ruok mntr
Zookeeper Monitoring - JMX • Mbeans Quorum ServerCnxn Leader LocalPeer Follower LeaderElection DataTree
Data Directory Cleanup • autopurge.snapRetainCount : Maintains n number of recent snapshots along with transaction log • autopurge.purgeInterval : Time Interval at which purge event will trigger
Zookeeper Future Releases - 3.5.X • Dynamic Reconfiguration w/o Rolling restarts • Separate Dynamic configuration file • Adding/Removing Servers • Modifying server parameters • Rebalance Client Connections • Backward Compatible • Modifying Server Roles[Obersver/Participants]
Zookeeper in Real World • Apache Kafka • Apache Storm • Apache HBase • Apache Solr • Apache Mesos • Apache Hive
Questions ? Thank You

More Related Content

Meetup on Apache Zookeeper

  • 2. What is distributed system ? • A distributed systems provides single coherent view of collection of independent computing machines. • Multiple software components on multiple computers, which run as a single system. • It can be locally or geographically connected.
  • 3. Characteristics of Distributed System • Concurrency • Fault Tolerance • Scalability • Transparency • Heterogeneity
  • 4. Delusion of Distributed System • Reliable Network • Zero Latency • Infinite Bandwidth • Secure Network • Topology Doesn’t change • Homogeneous Network • There is one administrator
  • 5. Introducing Zookeeper • Zookeeper provides coordination between distributed process via shared hierarchical namespace.
  • 6. Why Zookeeper ? • Naming Service • Locking • Synchronization • Configuration Management • Leader Election
  • 8. Zookeeper Architecture • Zookeeper follows client server architecture • Collection of Zookeeper servers is called an ensemble • At given time, a Zookeeper client is connected to one Zookeeper server • Client read requests are catered by connected Zookeeper server • Client write requests are forwarded to Zookeeper leader. • Write is considered successful, if quorum is achieved.
  • 9. Zookeeper Architecture • Client updates are sequentially consistent • Updates are atomic • Updates are reliable • Timeliness
  • 10. ZAB Protocol • Zookeeper Atomic Broadcast • Propagates state changes from leader to followers • Guarantees the state change
  • 11. Znodes • Organized similar to standard Unix and Linux file system • Similar to files and directories • Both directory and leaf znode holds data • Stores data <= 1MB
  • 12. Znode - Hierarchical Namespace
  • 13. Time in Zookeeper • Zxid : Every change to ZK state receives a stamp in form of zxid • Version Number : Change to Znode will cause change in Version • Ticks : Ticks define timing of events • Real Time : To put timestamp of stat structure and znode modification
  • 14. Znode Stat Structure • Czxid : Change that caused Znode to be created • Mzxid : Change that last modified Znode • Pzxid : Change that last modified children of the Znode • Ctime : Millisecond from epoch, when this Znode was created • Mtime : Millisecond from epoch, when this Znode was last modified • Version : Number of changes to data of Znode • Cversion : Number of changes to children of Znode • Aversion : Number of changes to ACL of Znode • EphemeralOwner : SessionId of owner of this Znode • DataLength : Length of data field • Numchildren : Number of children
  • 15. Zookeeper Operations create Creates Znode delete Deletes Znode exists Does Znode exist? getACL Get ACL permissions setACL Set ACL permissions
  • 16. Zookeeper Operations getChildren Gets the childrens of Znode getData Gets Data of Znode setData Sets Data of Znode Sync Syncs the client View
  • 17. Types of Znodes • Ephemeral Znode : Deleted with termination of client session, also can’t have any child znode. • Persistent Znode : Can have child znode, It is explicitly required to delete it. • Sequential Znode : Append a monotonicly increasing counter to the end of path. It is unique to parent znode.
  • 18. Znode Watchers • One time triggers • Sent to client which sets the watch • Maintained locally at zookeeper • Client won’t receive watches after getting disconnected • Two lists of watches – Data watches : create(),setData(), delete() – Child watches : create(), delete()
  • 19. Zookeeper ACL • ACL Permission – Create – Read – Write – Delete – Admin
  • 20. Zookeeper ACL • ACL Scheme – World – Auth – Digest – Host – IP
  • 21. Zookeeper Recipes/Higher Order Functions/Apache Curator • Naming Service • Service Discovery • Configuration Management • Locks • 2-phase commit • Leader Election • Barriers • Queues
  • 22. Netflix/Soabase Exhibitor • Zookeeper server is fail fast • Zookeeper is self-healing, (i.e. crashed/failed server if brought online, will automatically join the cluster) • Instance Monitoring • Log Cleanup • Backup/Restore • Cluster-wide configuration • Rolling Ensemble Change • Rich REST API
  • 23. Backup/Exhibitor • Online Backup - Use Netflix Exhibitor • Offline Backup - Bring down the old ensemble cluster and copy the zookeeper data directory to the new ensemble cluster
  • 24. Zookeeper Monitoring • 4 Letter Words • JMX
  • 25. Zookeeper Monitoring - 4 Letter Words conf srst cons srvr dump stat envi dirs (3.5.1) ruok mntr
  • 26. Zookeeper Monitoring - JMX • Mbeans Quorum ServerCnxn Leader LocalPeer Follower LeaderElection DataTree
  • 27. Data Directory Cleanup • autopurge.snapRetainCount : Maintains n number of recent snapshots along with transaction log • autopurge.purgeInterval : Time Interval at which purge event will trigger
  • 28. Zookeeper Future Releases - 3.5.X • Dynamic Reconfiguration w/o Rolling restarts • Separate Dynamic configuration file • Adding/Removing Servers • Modifying server parameters • Rebalance Client Connections • Backward Compatible • Modifying Server Roles[Obersver/Participants]
  • 29. Zookeeper in Real World • Apache Kafka • Apache Storm • Apache HBase • Apache Solr • Apache Mesos • Apache Hive