ScyllaDB Tablets: Rethinking Replication
•Download as PPTX, PDF•
0 likes•417 views
ScyllaDB is making a major architecture shift. We’re moving from vNode replication to tablets – fragments of tables that are distributed independently, enabling dynamic data distribution and extreme elasticity. In this keynote, ScyllaDB co-founder and CTO Avi Kivity explains the reason for this shift, provides a look at the implementation and roadmap, and shares how this shift benefits ScyllaDB users.
Recommended
More Related Content
Similar to ScyllaDB Tablets: Rethinking Replication
Similar to ScyllaDB Tablets: Rethinking Replication (20)
More from ScyllaDB
More from ScyllaDB (20)
Recently uploaded
Recently uploaded (20)
ScyllaDB Tablets: Rethinking Replication
- 1. ScyllaDB Tablets: Rethinking Replication Avi Kivity, CTO @ ScyllaDB
- 2. Avi Kivity ■ CTO, co-founder ScyllaDB ■ Linux KVM author and ex-maintainer
- 3. ■ Goals ■ Short Demo ■ The problem we’re fixing ■ Implementation details ■ Deep dive into Demo ■ Status Agenda
- 5. Goals ■ Fast bootstrap/decommission ■ Bootstrap is a critical time for a cluster if running out of space or CPU capacity ■ Incremental bootstrap ■ Shoulder the load immediately, not after bootstrap completes ■ Parallel bootstrap ■ Add multiple nodes in parallel if we’re in a real hurry ■ Decouple topology operations ■ Remove a dead node while bootstrapping two new nodes ■ Improve support for many small tables
- 6. Demo
- 7. ■ Scenario ■ Preload a 3-node cluster with 650 GB/replica ■ Run a moderate mixed read/write workload ■ Bootstrap three nodes ■ Decommission three nodes ■ Test harness ■ Scylla-cluster-tests (open-source) ■ Used in weekly regression tests Demo Scenario
- 10. History
- 11. How did we get here? ■ ScyllaDB streaming was fast… but node storage grew faster ■ Some schema shapes slow down mutation-based streaming ■ Eventually consistent, masterless architecture means the operator has to coordinate everything … ■ … and everything is serialized ■ Layout is static (famous Token Ring) A B A B Token ring For 2-node cluster After adding one node A B A B C C
- 12. Implementation
- 13. ■ Introduce a new layer of indirection - the tablets table ■ Each table has its own token range to node mapping ■ Mapping can change independently of node addition and removal ■ A new node can be added without any owned data! ■ Different tables can have different tablet counts ■ Tablet counts change so tablet size on disk remains roughly constant ■ Managed by Raft Group 0 Implementation - metadata System, tablets Query Replica Set Token
- 14. ■ Each tablet replica is isolated into its own memtable+sstables ■ Forms its own little Log-Structured Merge Tree ■ With compaction and stuff ■ Can be migrated as a unit ■ Migration: copy the unit ■ Cleanup: delete the unit ■ Split/merge as the table grows/shrinks Implementation - data path
- 15. ■ Hosted on one node ■ But can be migrated freely if the node is down ■ Synchronized via Raft ■ Collects statistics on tables and tablets ■ Migrates to balance space ■ Evacuates nodes to decommission ■ Migrates to balance CPU load ■ Rebuilds and repairs Implementation - load balancer
- 16. ■ Source of truth for the cluster ■ How many tablets for each table ■ Token boundaries for each tablets ■ On which nodes and shards do we have replicas ■ What kind of transition the tablet is undergoing ■ Which nodes and shards will host tablet replicas after the transition ■ Managed using the Raft protocol ■ Replicated on every node The tablets table CREATE TABLE system.tablets ( table_id uuid, last_token bigint, keyspace_name text static, replicas list<tuple<uuid, int>>, new_replicas list<tuple<uuid, int>>, session uuid, stage text, transition text, table_name text static, tablet_count int static, PRIMARY KEY (table_id, last_token) );
- 17. ■ Drivers are tablet-aware ■ But without reading the tablets table ■ Driver contacts a random node/shard ■ On miss, gets updated routing information for that tablet ■ Ensures fast start-up even with 100,000 entries in the tablets table Driver support
- 18. Deeper dive into demo
- 21. CPU load (per shard)
- 22. Disk bandwidth - streaming
- 24. Disk bandwidth - commitlog
- 26. Status
- 27. ■ What works ■ Parallel, incremental bootstrap and decommission ■ File streaming ■ Replace and rebuild ■ Space load balancing ■ Split on table growth ■ Materialized Views ■ Still working on it ■ Merge on table shrinkage ■ Change Data Capture ■ CPU load balancing ■ Automatic migration from Vnodes Status/plans
- 28. Stay in Touch Avi Kivity avi@scylladb.com @AviKivity @avikivity https://www.linkedin.com/in/avikivity/
- 29. Graphics Sandbox
- 30. Illustration - request looks up into a table of tablets, exits lookup with (node, shard) routing information Where is this tablet? Node B Shard 3 system.tablets Query Replica Set
- 35. Vnodes, advanced
- 37. A node in which there are 16 equally sized slots for tablets, only some are filled. In each filled slot there is a memtable + a few sstables Memtable SSTable SSTable Memtable SSTable SSTable SSTable Memtable SSTable SSTable SSTable SSTable SSTable SSTable SSTable Memtable SSTable SSTable Memtable SSTable Memtable SSTable SSTable SSTable Memtable SSTable Memtable SSTable SSTable SSTable SSTable Memtable SSTable SSTable Memtable SSTable
- 38. Illustration - maybe an octopus spreading its tentacles to each node Maybe use DB Performance at Scale book style tentacles here?
- 39. Illustration - Deuteronomy style tablets but with host/shard information instead of commandments
- 40. Felipe's replacement idea for deuteronomy thingie CREATE TABLE system.tablets ( keyspace_name text static, table_name text static, tablet_count int static, table_id uuid, last_token bigint, … PRIMARY KEY(table_id, last_token) ) Raft-Managed Table User Table Replicas