Distributed Systems

Time and Global States

MISM 95-702 Distributed Systems 1

 Lab Debrief Q1
In the JMS Lab, what difference is there
between JMS and Message Oriented
Middleware (MOM)?
a) None, they're the same.
b) MOM provides an interface to JMS.
c) JMS provides Point-to-Point Queues and
Topics, but MOMs only provide Topics.
d) JMS only interacts with Servlets.
e) JMS provides an interface to a MOM

MISM 95-702 Distributed Systems 2

 Lab Debrief Q2
In the JMS Lab, can a module be *both* a queue
listener and a queue writer?
a) No, because only one connection is allowed.
b) Yes, because every queue needs both.
c) No, because the JMS protocol only allows one or
the other.
d) Yes, that's how data is transferred from one
queue to another.
e) No, that's only possible with Topics.

MISM 95-702 Distributed Systems 3

 Learning Goals
• To understand:
– The challenge of time in a distributed
system
– How to synchronize distributed clocks
– How you can assess the state of a
distributed system

MISM 95-702 Distributed Systems 5

 Skew and drift
• Why can’t we have a global clock in a
distributed system?
• Because clocks are imperfect:
– Clock skew – different times
• Two clocks will have two different times.
– Clock drift – different speeds
• Each clock varies in speed

MISM 95-702 Distributed Systems 6

 Time
• What is a second?
– 9,192,631,770 periods of transition between the two
hyperfine levels of the ground state of Caesium-133
(Cs133)

• Ordinary quartz crystal clocks

– Drifts 1 second every 11 days

• What can a computer do in a second?

– Intel Core i7: ~100 Billion Instructions

MISM 95-702 Distributed Systems 7

 Clocks
• Cesium clocks
– Expensive
• GPS receiver
– Less expensive
– (GPS system has cesium clock(s))

MISM 95-702 Distributed Systems 8

 Network Time Protocol
Design Goals:
• Sync with UTC over
Internet
• Provide reliability via
redundancy
• Scale to large number
of clients and servers
• Defend against Mallory Graphic source:
http://en.wikipedia.org/wiki/Network_Time_Protocol

MISM 95-702 Distributed Systems 10

 NTP Simulation
• Work in pairs
• Both students should have the clock demo page running
– http://tinyurl.com/702clock
– Use a laptop, not a phone, and don’t allow the browser to sleep
• Use only ONE form sheet
• ONE student starts the process
– fill in (a) and pass the form to student TWO
• student TWO
– fills in (b) upon receipt of the form
– then fills in (c) and passes the form back to student ONE
• student ONE
– fills in (d)
– calculates (e-i)
– puts the value of (i) in their own clock demo (ONE only)
• Both students
– eyeball your clocks and see if they are nearly synchronized 11
UDP Packet (reusable) UDP Packet (reusable) UDP Packet (reusable)

a Sent time a Sent time a Sent time

b Received time b Received time b Received time

c Sent-back time c Sent-back time c Sent-back time

Returned-back Returned-back Returned-back

d d d
time time time

Calculation Calculation Calculation

Total round trip Total round trip Total round trip
e time e time e time
(d-a) (d-a) (d-a)
Remote processing Remote processing Remote processing
f f f
time (c-b) time (c-b) time (c-b)
Delay each way Delay each way Delay each way
g g g
(e-f)/2 (e-f)/2 (e-f)/2
Offset relative to Offset relative to Offset relative to
h remote h remote h remote
(d-g) - c (d-g) - c (d-g) - c
Amount to adjust Amount to adjust Amount to adjust
i local clock i local clock i local clock
-h -h -h

MISM 95-702 Distributed Systems 12

 NTP

NTP b c
Server

Laptop
a d

Time

MISM 95-702 Distributed Systems 13

 Summarize
• Summarize in your own words how
NTP synchronization works
• What is NTP synchronized time good
enough for?
– Median error is 2-5ms but has a long tail
• Source: Chi-Yao Hong , Chia-Chi Lin , Matthew Caesar, Clockscalpel:
understanding root causes of internet clock synchronization inaccuracy,
Proceedings of the 12th international conference on Passive and active
measurement, p.204-213, March 20-22, 2011, Atlanta, GA

• What are its shortcomings?

MISM 95-702 Distributed Systems 14

 Synchronization
• There are two ways to synchronize
clocks in a distributed system.

1. Internal Synchronization:
– Peer systems synchronize among
themselves
– Bound
• If the synchronization scheme has a bound D
– (i.e. two clocks will have max D skew)
• Then all clocks will agree within D.
MISM 95-702 Distributed Systems 15
 Synchronization
2. External Synchronization
– Systems all synchronize with a more
authoritative external clock
– I.e. UTC (Coordinated Universal Time)
– Bound:
• Each clock is accurate to UTC within D

MISM 95-702 Distributed Systems 16

Process Histories Global History
State 3c, 6p

& Global Histories SendA1 B:2p

State 3c, 4p
Process A
State 4c, 6p
State 3c, 6p
Process B RecA1 A:2p
SendA1 B:2p
State 4c, 6p State 4c, 8p
State 3c, 4p
RecA1 A:2p SendB1 A:1c
RecB1 B:1c
State 4c, 8p …
State 4c, 4p
SendB1 A:1c
SendA2 B:2c
State 3c, 8p
State 2c, 4p
RecA3 A:2p Global Histories are:
SendA3 B:2p
State 3c, 10p • causal
State 2c, 2p • conjectures
SendB2 A:2p
RecB2 B:2p • they are not physical
State 3c, 8p
State 2c, 4p

MISM 95-702 OCT 17

 Consistent histories
Global History
Process A
State 4c, 6p
State 3c, 6p Before Process B
State 3c, 6p
SendA1 B:2p State 4c, 6p
State 3c, 4p RecA1 A:2p
After
RecB1 B:1c State 4c, 8p
State 4c, 4p SendB1 A:1c
SendA2 B:2c State 3c, 8p
State 2c, 4p RecA3 A:2p
SendA3 B:2p State 3c, 10p
State 2c, 2p SendB2 A:2p
RecB2 B:2p State 3c, 8p
State 2c, 4p

MISM 95-702 OCT 18

 Consistent histories
Global History
Process A
State 3c, 6p
State 3c, 6p Before Process B
State 4c, 6p
SendA1 B:2p State 4c, 6p
State 3c, 4p RecA1 A:2p
After
RecB1 B:1c State 4c, 8p
State 4c, 4p SendB1 A:1c
SendA2 B:2c State 3c, 8p
State 2c, 4p RecA3 A:2p
SendA3 B:2p State 3c, 10p
State 2c, 2p SendB2 A:2p
RecB2 B:2p State 3c, 8p
State 2c, 4p

MISM 95-702 OCT 19

 Consistent histories
Global History
Process A
State 4c, 6p
State 3c, 6p Process B
State 3c, 6p
SendA1 B:2p Before State 4c, 6p
SendA1 B:2p
State 3c, 4p RecA1 A:2p
After
RecB1 B:1c State 4c, 8p
State 4c, 4p SendB1 A:1c
SendA2 B:2c State 3c, 8p
State 2c, 4p RecA3 A:2p
SendA3 B:2p State 3c, 10p
State 2c, 2p SendB2 A:2p
RecB2 B:2p State 3c, 8p
State 2c, 4p

MISM 95-702 OCT 20

 Consistent histories
Global History
Process A
State 4c, 6p
State 3c, 6p Process B
State 3c, 6p
SendA1 B:2p Before State 4c, 6p
SendA1 B:2p
State 3c, 4p RecA1 A:2p
State 3c, 4p
RecB1 B:1c State 4c, 8p
After
State 4c, 4p SendB1 A:1c
SendA2 B:2c State 3c, 8p
State 2c, 4p RecA3 A:2p
SendA3 B:2p State 3c, 10p
State 2c, 2p SendB2 A:2p
RecB2 B:2p State 3c, 8p
State 2c, 4p

MISM 95-702 OCT 21

 inConsistent histories
Global History
Process A
State 4c, 6p
State 3c, 6p Process B
State 3c, 6p
SendA1 B:2p Before State 4c, 6p
SendA1 B:2p
State 3c, 4p RecA1 A:2p
RecB1 B:1c X State 4c, 8p
State 3c, 4p
RecB1 B:1c
State 4c, 4p SendB1 A:1c
SendA2 B:2c
After
State 3c, 8p
State 2c, 4p RecA3 A:2p
SendA3 B:2p State 3c, 10p
State 2c, 2p SendB2 A:2p
RecB2 B:2p State 3c, 8p
State 2c, 4p

MISM 95-702 OCT 22

 Consistent histories
Global History
Process A
State 4c, 6p
State 3c, 6p Process B
State 3c, 6p
SendA1 B:2p State 4c, 6p
SendA1 B:2p
State 3c, 4p Before RecA1 A:2p
State 3c, 4p
RecB1 B:1c State 4c, 8p
RecA1 A:2p
State 4c, 4p SendB1 A:1c
State 4c, 8p
SendA2 B:2c After State 3c, 8p
State 2c, 4p RecA3 A:2p
SendA3 B:2p State 3c, 10p
State 2c, 2p SendB2 A:2p
RecB2 B:2p State 3c, 8p
State 2c, 4p

MISM 95-702 OCT 23

 Useful terminology to know
• Process history
• Happened-before relation
• Consistent Global history
– A linearization of process histories
– Maintaining the happened-before relation
• Consistent Global State
– Also called a cut
– A set of process states that could have occurred at the
same time.
– Consistent cut
• A cut that maintains the happened-before relation
– Inconsistent cut
• Breaks the happened-before relation

• Read Coulouris for concise definitions

– And know definitions for final exam
– Wikipedia can also be helpful

MISM 95-702 Distributed Systems 24

 Diagrams can help to reason
about global histories
p1
a b m1

p2 Physical
c d time
m2

p3
e f

MISM 95-702 OCT 25

 Consistent Cuts Example

p1
a b m1

p2 Physical
c d time
m2

p3
e f

Because no global clock

a straight line shows more
accuracy than really possible
Source: Distributed Computing: Principles, Algorithms, and Systems By Ajay D. Kshemkalyani, Mukesh Singha
MISM 95-702 OCT 26
 Are these global states (cuts) consistent?

p1
a b m1

p2 Physical
c d time
m2

p3
e f

Notice state would include p1-3 but also m1

MISM 95-702 OCT 27

 Does a->e?

p1
a b m1

p2 Physical
c d time
m2

p3
e f

MISM 95-702 OCT 28

Logical vs Physical Concurrence
• Logical concurrence: In distributed systems, two
events are logically concurrent if they do not
causally affect each other.
• Physical concurrence:
p
connotes that two
1
a b m 1

events happen at
p Physical
the same instance 2
c d m
time
2
in time.
p3
– This is unknowable in e f
a distributed system.
• e is logically concurrent to all of a, b, m1, c, d, m2
• f is not logically concurrent to any of the events

MISM 95-702 OCT 29

 Lamport (Logical) Clocks
• An alternative to (skewed, drifting) physical clocks
• Two rules for assigning logical times to events:
1. Events on one process happen in order
• Each happens-before the next
• Increment the logical clock by 1 for each event
2. The passing of messages can be used to indicate
happens-before between processes
• The sending of the message happens-before the receiving of the
message.
• Take the larger of the logical clock (+1) value from the process or the
message

MISM 95-702 OCT 30

 Lamport Clock numbering
1 2
p1
a b m1

3 4
p2 Physical
time
c d
m2

1 5
p3
e f

MISM 95-702 OCT 31

 What time is g?
1 2
p1
a b m1

3 4
p2 Physical
time
c d
m2

1 5
p3
e g f

MISM 95-702 OCT 32

 Now what time is g?
1 2
p1
a b m1

3 4
p2 Physical
time
c d
m2

1 5
p3
e g f

MISM 95-702 OCT 33

 L(d)>L(g)
so did d happen after g?
1 2
p1
a b m1

3 4
Physical
p2
c d time
m2

1 2 5
p3
e g f Is this still useful?
No. - It gives a partial ordering.
- d->f implies L(d) < L(f) - The Lamport Clocks of two
causally-related events
- L(g) < L(d) does not imply g->d
will be ordered
- In this system design, d and g
are not causally related.
- It does not matter if g->d
MISM 95-702 OCT 34
 Total & Causal Ordering
• Lamport Clocks can be used to coordinate the ordering of
distributed events so that each process executes events in
the same order (e.g. updating a database)
• Vector clocks are extensions to Lamport's work:
– Each process is assigned a process-index
– Messages between processes carry a vector of process-index and
Lamport-clock with them
• E.g. Dynamo, Amazon.com’s highly available key-value
storage system, uses vector clocks.
– See: http://s3.amazonaws.com/AllThingsDistributed/sosp/amazon-dynamo-sosp2007.pdf Section 4.4

MISM 95-702 Distributed Systems 35

 Global State Problem 1
• We have stores and warehouses all
over the world
• Each has a local system that tracks
inventory.
• What is our current level of inventory?

MISM 95-702 Distributed Systems 36

 Global State Problem 2
• We have a very complex chemical
manufacturing plant
• Each sensor and valve is computer
controlled
• There are some sensor and valve
combinations that are very dangerous
• How do we know if we are in one of
those states?

MISM 95-702 Distributed Systems 37

 Finding Global States
• It is often important to obtain the global
state of a distributed system.
• It would be nice to
– Be omniscient – see all at once
– Or stop time
• But in distributed systems, we only have
unreliable messages that take time to go
from one process to another.
• What to do?

MISM 95-702 Distributed Systems 38

 Chandy & Lamport
Snapshot Algorithm
• Assumptions:
– Reliable channels
• A channel is a network path from one process to
another
– Every message sent is reliably received
• Once (not more than once)
• In order (FIFO)
– There is a channel (path) between every 2
processes
– Any process may initiate a snapshot
– Processes continue as normal while snapshot is
taking place.
• They don't stop their work!
MISM 95-702 Distributed Systems 39
 Snapshot Algorithm
• Marker receiving rule:
When I receive a marker on channel c
If (I am not yet taking part in this snapshot) {
Record my state now (count # of each commodity in hand only)
Do Marker sending rule
Record that the state of channel c is empty
Begin keeping track of state all other incoming channels (except c):
(I.e. count commodities arriving via each channel.)
} else if this marker is coming in from a new channel c {
Record that the state of channel c is all messages it has received since it began
keeping track. (No more coming.)
Stop recording any further messages on channel c (they are now post-shapshot)
If (this is last channel to receive marker on) send records to Monitor process
}
• Marker sending rule:
For (each outgoing channel c) {
Send one marker message over c
}

MISM 95-702 Distributed Systems 40

 Count the total of c's and p's
Process A
State 3c, 6p Process B
SendA1 B:2p State 4c, 6p
A B Tot
State 3c, 4p RecA1 A:2p
9 10 19
RecB1 B:1c State 4c, 8p
State 4c, 4p SendB1 A:1c
SendA2 B:2c State 3c, 8p
State 2c, 4p RecA3 A:2p
SendA3 B:2p State 3c, 10p
State 2c, 2p SendB2 A:2p
RecB2 B:2p State 3c, 8p
State 2c, 4p

MISM 95-702 OCT 41

 Count the total of c's and p's
Process A
State 3c, 6p Process B
SendA1 B:2p State 4c, 6p
State 3c, 4p RecA1 A:2p A B Tot
Net Tot
RecB1 B:1c State 4c, 8p 7 10 17
2 19
State 4c, 4p SendB1 A:1c
SendA2 B:2c State 3c, 8p
State 2c, 4p RecA3 A:2p
SendA3 B:2p State 3c, 10p
State 2c, 2p SendB2 A:2p
RecB2 B:2p State 3c, 8p
State 2c, 4p

MISM 95-702 OCT 42

 Remember: Snapshot state
needs to account for messages
in progress
p1
a b m1

p2 Physical
c d time
m2

p3
e f

Source: Distributed Computing: Principles, Algorithms, and Systems By Ajay D. Kshemkalyani, Mukesh Singha
MISM 95-702 OCT 43
 C12

P1 P2

C21

Y B R Y B R
P1 P2
C21 C12
Total Total
 Y B R
P1 P1
C21
C12
C31
TOTAL
C13

C31
C21
P3 P2

C32

C23
Y B R Y B R
P3 P2
C13 C12
C23 C32
TOTAL TOTAL
 PITplayer1
Queue: jms/PITplayer1

Queue: jms/PITplayer2

PITplayer3 PITplayer2

Queue: jms/PITplayer3
In the trading simulation, the channels are implemented as JMS Queues.
This meets the Chandy & Lamport snapshot algorithm assumption that
there are channels from each player to every other player, and that the
channels are First-In-First-Out.
All players are implemented as Message Drive Beans (aka Queue
Listeners)
Using State for Debugging
• A: waitreply(B) • How could info
• B: like this be
• C: waitreply(A) collected?
• D: waitreply(F)
• E: waitreply(F) • What does it say
about the state of
• F: waitreply(G) the system?
• G: waitreply(D)

MISM 95-702 OCT 47

Global State for Termination
• A: transaction 3382 completed
• B: transaction 3382 completed
• C: transaction 3382 incomplete
• D: transaction 3382 completed
• E: transaction 3382 completed

MISM 95-702 Distributed Systems 48