Practical Systems

Among the most famous systems built on consensus algorithms are chubby and zookeeper. Chubby[2] is developed in google as a distributed coarse grained lock service, which has been adopted by many systems like GFS, Bigtable, etc. Chubby provides a filesystem-like api for user. Users can lock files and store information in them. It provides strong consistence to users, which means that read will always return update data. Though this approach can simplify usage of chubby client, its read performance is limited.

Zookeeper[1] is an open source replicated state machine built on an consensus algorithm called zab, which is quite similar to raft. Zookeeper provides a similar interface to chubby: filesystem-like api, store a small amount of information in files, etc. However, zookeeper provides a quite different semantics. The authors of zookeeper realizes that zookeeper will be used in high read-write ratio workload, where read performance is quite important. Zookeeper client's read request is performed in local server, no matter whether this client is connected to a master or follower. This means that client may read stale data unless client sends sync request before read.

At last, I have implemented raft on a system called logmaster using akka finite state machine module. This is not a complete system like zookeeper or chubby. It's just a prototype implementation of raft protocol. The main goal of this system is to help understanding raft, so I don't even provide an implementation to persist logs. Thanks to akka's great abstraction, I don't need to care about anything except the algorithm. These are two lessons I've learned some lessons when implementing logmaster:

  1. You should write out code even if you feel that you've understood an algorithm thoroughly. It can help you achieve a better grasp of this algorithm.
  2. It's nontrivial to build a distributed system based on consensus algorithm, even if the algorithm is quite easy to understand. There are many things to consider, disk failure, client communication, session management etc.

References

  1. Yahoo. ZooKeeper: Wait-free coordination for Internet-scale systems.
  2. Mike Burrows. The Chubby lock service for loosely-coupled distributed systems.