The document discusses the evolution of distributed systems from single machines to replicated databases and services. It explains how eventual consistency allows for higher availability but reduces correctness by allowing stale data reads. The key is that different applications have different consistency needs based on their risk tolerance. Rather than strict consistency, eventual consistency with apologies is often sufficient and enables more flexible tradeoffs between correctness and availability for increased business value.
2. About Me
• Work at ThoughtWorks Pune
• DDD and Distributed Computing enthusiast
• A fan of Pat Helland
Twitter handle: @shripadagashe
Blog: https://shripad-agashe.github.io
3. Evolution of systems
Image source: https://commons.wikimedia.org/wiki/File:Front_Z9_2094.jpg
Image Source: https://en.wikipedia.org/wiki/Solaris_Cluster#/media/
File:Sun_Microsystems_Solaris_computer_cluster.jpg
Vs
4. Reliability Is a steep curve
Downtime vs % availability
0
225
450
675
900
99 99.9 99.99 99.999
Downtime in Seconds
%Availability Per Year
Per
month
Per day
99 3.65 days 7.2 hour
14.4
minutes
99.9
8.76
hours
43.8
minutes
1.44
minutes
99.99
52
minutes
4.38
munutes
9 seconds
99.999
5.26
minutes
25.9
seconds
0.8
seoncds
5. And It has a price
0
250
500
750
1000
99 99.9 99.99 99.999
6. Probability theory to the
rescue
Union of Probability
P(A) Intersection P(B) = P(A) * P(B)
For 99 % Availability i.e. 1% unavailability Probability of unavailability for 2 servers
= (0.01) * (0.01) = 0.0001
7. More on Probability
• Systems in series
• Availability = P(A) * P(B)
• Systems in parallel
• Availability = 1 - P(1-A) * P(1-B)
8. So Architectural patterns
evolve around it
App App
DB
Box Cylinder Architecture Best Practices
•App layer should be
stateless
• Architecture should be
layered
9. But DB is still on a single
machine
Primary DB
Secondary
DB
Asynchronous
replication
DB high availability is achieved via replication
Active-Active Replication Active-Passive Replication
Primary DB
Secondary
DB
Synchronous
Replication
10. Move to transaction model
Client In Memory Disk
Write
Write
Commit
Write to Disk
15. Bouquets and brickbats
+
• Integration is simple
• Familiar for most
developers
• Easier to reason
-
• Any sync call will add
to latency
• Sync calls will
expose system to
variations in behavior
of external systems
17. Bouquets and brickbats
+
• As there is no sync call, it
does not add additional
latency to app
• As systems are isolated
chances of failure
propagation are minimal
• With Pub Sub changes can
be propagated to multiple
subscriber with minimal
additional work
-
• Integration may not be
trivial
• Async propagation of data
needs careful reasoning
18. Probabilistic Business Rules
• When we have asynchronous replication we have
windows of failure that mean work may be lost or
delayed.
• Distribution + AsynchronyàProbabilities of
Enforcement
Source:http://db.cs.berkeley.edu/cs286/papers/quicksand-cidr2009.pdf
20. Here comes Eventual
Consistency
• Eventual consistency guarantees that subset of previous
writes will be returned; eventually it will return all
writes.
• There is no guarantee of order
• There is no time bound on eventual
• Loosely defined term which guarantees nothing. The
application should tolerate any subset of writes
without any time guarantee
• As opposed to EC being a single concept it is a spectrum
• On one end of spectrum is strong consistency
• On the other end eventual consistency
21. Eventual consistency thru simple
example
Official scorekeeper:
score = Read (“visitors”);
Write (“visitors”, score +1);
Umpire:
if middle of 9th
inning then
vScore = Read (“visitors”);
hScore = Read (“home”);
if vScore < hScore
end game;
Radio reporter:
do {
vScore = Read (“visitors”);
hScore = Read (“home”);
report vScore and hScore;
sleep (30 minutes);
}
Sportswriter:
While not end of game {
drink beer;
smoke cigar;
}
go out to dinner;
vScore = Read (“visitors”);
hScore = Read (“home”);
write article;
Statistician:
Wait for end of game;
score = Read (“home”);
stat = Read (“season-runs”);
Write (“season-runs”, stat +
score);
Stat watcher:
stat = Read (“season-
runs”);
discuss stats with friends;
Strong Consistency See all previous writes.
Eventual Consistency See subset of previous writes.
Consistent Prefix See initial sequence of writes.
Bounded Staleness See all “old” writes.
Monotonic Reads See increasing subset of writes.
Read My Writes See all writes performed by reader.
Source:http://cacm.acm.org/magazines/2013/12/169945-replicated-data-consistency-explained-through-baseball/fulltext#F8
22. Not everyone needs same
thing
• Often different roles have different tolerances for
stale information
• The trade off between correctness and availability
can bring in more revenue
• The trade off is often driven by business value
23. Whats the Risk Appetite
• Consistency is often cost of doing business
• The major point is that availability (and its cousins offline and
latency-reduction) may be traded off with classic notions of
consistency. This tradeoff may frequently be applied across many
different aspects at many levels of granularity within a single
application.
• Locally clear a check if the face value is less than $10,000. If it exceeds
$10,000, double check with all the replicas to make sure it clears
• Schedule the shipment of a “Harry Potter” book based on a local
opinion of the inventory. In contrast, the one and only one Gutenberg
bible requires strict coordination!
Source:http://db.cs.berkeley.edu/cs286/papers/quicksand-cidr2009.pdf
24. Memories, Guesses,
Apologies
• The idea is that everything is done locally with a
subset of the global knowledge.
• You know what you know when an action is
performed. Since you have only a subset of the
knowledge, your actions are really only guesses.
• When your knowledge as a replica increases, you
may have an “Oh, crap!” moment.
Source:http://db.cs.berkeley.edu/cs286/papers/quicksand-cidr2009.pdf
25. • Every business has to be ready for apologies.
• Consider a case where the only book in inventory is
scheduled for delivery.
• In preparing the book for shipment, it is run over by the
forklift in the warehouse.
• So correct software non withstanding you will need
to apologize
More on Apologies
Source:http://db.cs.berkeley.edu/cs286/papers/quicksand-cidr2009.pdf
26. How to apologize
• First of all recognize if you need to apologize
• Identify promises that could not be completed
• Unique identifier across systems becomes critical to
identify failures
• Typically the mistakes are identified during
reconciliation
• Based on severity apology can be handled by system
based on rules or directed for human involvement
27. What inhibits business trade offs
• The layering of an arbitrary application atop a storage subsystem
inhibits reordering (and also apologies)
• Logical delete vs Actual delete of row
• Only when commutative operations are used can we achieve the
desired loose coupling.
• Application operations can be commutative
Source:http://db.cs.berkeley.edu/cs286/papers/quicksand-cidr2009.pdf
28. Commutative Business Transactions
• Order insensitive logic
• Valid Account creation - Create dummy account first and then
attach customer to it later
• Credit to account
• Visibility of business operations
• Logical deletion of record
30. We need to partner with
people like him
Image source: https://commons.wikimedia.org/wiki/File:Jackie_Stewart_2011_British_Grand_Prix.jpg
31. So whats in it for You
• The technique explained requires business person
to be IT sympathetic
• Business has to align with IT and see IT as a
competitive advantage
• Developing with us rather than developing for us
mentality