Author: Linghan Xing, Junyi Wang Date: 4/10/2017
This is an implementation of a distributed key value store system.
When a put
request is generated by the client
, server
will generate a
randomised algorithm to decide which node to store the data in, the idea is
that due to randomization, the load of local storage should be distributed
evenly accross all available nodes.
After the target node
is decided, server
will send both key
and value
to the node
and expect to get a successful
response.
The get
request works in a similar way. The algorithm implementation ensures
that for each unique key
, the target node
remains the same.
A pthread_mutex_lock
was employed in the put
section to protect the hash
table when contention happens
When a client request to add
a node, it sends the node_info
to server, which
will decide where in the ring
would the new incoming node go. Once the node is
added to the ring
, the next
node in the ring will send all of its key-value
pairs to the server to ask for a re-distribution
of the keys, thus make sure
the consistent view of the system.
When a client request to drop
a node, it sends the node_info
to server, which
will signify the node to be removed to send
all of its key-value stores to the
server to re-distribute
them, thus make sure the consistent view of the system.
$ make
use$ ./testscript.sh
for quick demo.- First bring the
node
online since it's where all the data is supposed to be by$ ./node {port} localhost:3344
, eg.$ ./node 3345 localhost:3344
,$ ./node 3346 localhost:3344
,$ ./node 3347 localhost:3344
- Then bring
server
online by putting how many nodes are available and their ip addressed, for example$ ./server 3 localhost:3345 localhost:3346 localhost:3347
- Then the
client
could test using$ ./client localhost:3344 {method} key value
, note that thelocalhost:3344
is the IP address of server - Available methods:
put
add
get
drop
, note that node address should be provided instead of key-value pair in the case ofadd
anddrop
.
- [Bryant & O'HALLARON]: Computer Systems: a programmer's perspective
- Beej's guide for network programming
- Operating systems: three easy pieces
- cs.yale.edu for the handy hashing method
Enhance data stability/security