Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to content

A simple distributed storage system with consistent hashing

Notifications You must be signed in to change notification settings

LinghanX/key-store

Repository files navigation

A distributed key-value store system

Author: Linghan Xing, Junyi Wang Date: 4/10/2017

This is an implementation of a distributed key value store system. When a put request is generated by the client, server will generate a randomised algorithm to decide which node to store the data in, the idea is that due to randomization, the load of local storage should be distributed evenly accross all available nodes.

After the target node is decided, server will send both key and value to the node and expect to get a successful response. The get request works in a similar way. The algorithm implementation ensures that for each unique key, the target node remains the same.

A pthread_mutex_lock was employed in the put section to protect the hash table when contention happens

Implementation decisions

Consistent hashing

Add a node

When a client request to add a node, it sends the node_info to server, which will decide where in the ring would the new incoming node go. Once the node is added to the ring, the next node in the ring will send all of its key-value pairs to the server to ask for a re-distribution of the keys, thus make sure the consistent view of the system.

Drop a node

When a client request to drop a node, it sends the node_info to server, which will signify the node to be removed to send all of its key-value stores to the server to re-distribute them, thus make sure the consistent view of the system.

Set up server, node, and client

  1. $ make use $ ./testscript.sh for quick demo.
  2. First bring the node online since it's where all the data is supposed to be by $ ./node {port} localhost:3344, eg. $ ./node 3345 localhost:3344, $ ./node 3346 localhost:3344, $ ./node 3347 localhost:3344
  3. Then bring server online by putting how many nodes are available and their ip addressed, for example $ ./server 3 localhost:3345 localhost:3346 localhost:3347
  4. Then the client could test using $ ./client localhost:3344 {method} key value, note that the localhost:3344 is the IP address of server
  5. Available methods: put add get drop, note that node address should be provided instead of key-value pair in the case of add and drop.

helpful resources that made this happen

  1. [Bryant & O'HALLARON]: Computer Systems: a programmer's perspective
  2. Beej's guide for network programming
  3. Operating systems: three easy pieces
  4. cs.yale.edu for the handy hashing method

to do

Enhance data stability/security

About

A simple distributed storage system with consistent hashing

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published