A Leaner, More E ciemt, Available Copy Protocol

Darrell D. E. Long+ Department of Computer Science University of California Santa Cruz, CA 95064 Jehan-Frangois Piiris Department of Computer Science University of Houston Houston, TX 77204-3475

darrell @ cs.ucsc.edu
Abstract
Available copy protocols provide the highest data availability and data reliability of all replication protocols that do not regeneratefailed replicas. Unfortunately,all existing implementationsof available copy protocols either rely on complex procedures for ascertaining which replicas are up to date after a totalfailure or have to wait for the recovery of all failed sites. We present a simple technique for eficiently implementing the available copy protocol. Our protocol does not require version numbers and maintains only n + log(n) bits of state per replica. We also show under standard Markovian assumptions that our new protocol provides the same data availability as the best feasible implementationsof the available copy protocol.
Keywords: distributed $le systems, data replication, replication control protocols

paris @ cs. uh.edu

A first class of protocols makes the assumption that network partitions are either unlikely or unlikely to occasion conflicting updates. The best known of them are the avuilable copy protocol (AC) [2, 71, the regeneration algorithm [17] and the Coda replication control protocol [ 183.

The second class of protocols take the approach that data consistency is much more important than data availability. These protocols rely on quorums to provide mutual exclusion and prevent conflicting updates. As a result, they provide lower data availabilities than the other protocols. The best known quorum-orientedprotocols include majority consensus voting (MCV), weighted voting (WV) [ 5 ] , dynamic voting (DV) 141, dynamic-linear voting (DLV) [9] and voting with witnesses (VWW) [151. A common feature of all replication control protocols is the use of metadata to record the states of the replicas. These metadata nearly always include a version number, that is an integer that is incremented each time the replicated data are modified. Protocols such as optimistic available copy [ 103 and all dynamic voting protocols also require each replica to keep track of the identitiesof the replicas it believes to be operational. This informationis kept in a metadata structure, variously called a was-available set, a connection vector or a majority block. Despite the important role played by these metadata, the problem of finding the most efficientmetadata organization for a given replication control policy has not received the attention that it deserves. As we will see, the results of this neglect have been replication controlprotocols with bloated metadata and complex procedures for ascertaining which replicas are up to date. We present anew implementationof Bernstein and Goodmans available copy protocol [21. Our new protocol maintains for each replica a cohort set that is updated any time a failure is detected or a replica residing on a site that failed

1. Introduction
Ciia data are often replicated either to reduce read rtcl access times or to provide constant data availability in the presence of failures. This technique is known as data replication. As can be expected, data replication introduces its own problems, the most important of which is maintaining all replicas in a consistent state. This is a complex task because host failures and network partitions may occasion incomplete updates that leave some replicas inconsistent. Special replication control protocols have been devised to perform this task in a transparent fashion. These protocols differ in their message overhead, their handling of network partitionsand the data availabilitiesthey provide.
t The work of this author was done while a Visiting Scientisi ai IBM Almaden Research Center.

0-8186-7683-3/96 $05.00 0 1996 IEEE

400

is repaired. By requiring that all changes in the cohort set involve all sites in the new cohort set, we guarantee that all replicas sharing the same cohort set are identical and remove the need for maintaining version numbers. As a result, our protocol requires only n + log(n) bits of metadata per replica, that is n bits for storing the cohort set and log(n) bits for storing the identity of the replica The recovery procedure is also greatly simplifiedas it suffices now to gather all the replicas in any mutually agreed cohort set to find the current version of the replicated object.
The remainderof this paper is organizedas follows: Section 2 contains a review of existing replication control protocols and Section 3 introduces our new protocol; Section4 includes a study of the dependability of our protocol. Possible extensions are discussed in Section 5 while Section 6 has our conclusions.

excluded. An included site s is one that is known to hold a current replica of the data object while an excluded site t is one that has failed and whose failure has been recorded by an operational site executing an exclude(s) transaction. When a failed site t repairs following a failure, it attempts to locate another site s thait is operational. If such a site can be found, then t will repair from s and request s to execute the transaction include(t),, In the presence of a total failure, the sets of included and exxluded sets are used to determine the site-or set of sites-that failed last and holds a current replica of the data object.
2.2. The naive availablle copy protocol
The nai've available copy (NAC) protocol [16] avoids the problem of failure detection by not maintaining any site failure information. It behaves like the original available copy protocol except in the event of a total failure, in which case itmust wait for all sitesparticipatingin thereplicationto recover. The only metadatamaintainedby the NAC protocol are the version numbers of the replicas. The price for the simlplicity of the NAC protocol is a slower recovery after a total failure and a lower overall data availability. In most cases, total failures will be rather exceptional events that are much more likely to result from a catastrophic event affecting all sites holding replicas than from successive site failures. When this is the case, all available copy protocols will have to wait for the recovery of all sites holding replicas.

2. Available Copy Protocols

Availablecopy protocolsare based on the observationthat if any one site has received all updates to a given data object it holds the current version of the data object. Since they discount thepossibilityofnetworkpartitions,they can allow access to a replicated data object as long as a single replica of the data object remains available. As a consequence of this, they provide the highest data availability and data reliability of all replication protocols that do not regenerate failed replicas [161. There are three parts to an availablecopy protocol: write, read and recovery. The rule for writing is extremely simple: write to all accessible replicas. Since all accessible replicas receive each write, they are kept in a consistent state: the replicated data can then be read from any accessible replica. When a site holding a replica recovers from a failure, this replica needs to be compared, in some manner, with another replica that contains the current version of the data object. If all sites holding replicas of the data object have failed, no replica can recover until the last site(s) to fail can be found. This is the most complex part of any available protocol and the only one to differ significantlyfrom one implementation to another.

23. The optimistic available copy protocol

Like the original available copy protocol, the optimistic available copy (OAC) pnotoCol[ 101 maintains availability informationabout each and every replica but it only updates this informationwhen the replicated data object is modified or when a recovery occuirs. The protocol maintains two pieces of informationat each site holding a rcplica: a version number and a was-availableset. The was-availableset for an active replica s, denoted W,, lists those replicas that s knows to be up to date. This includes all replicas that received the most recent write and all replicas that have repaired from s since the last write. Was-available sets can be maintained inexpensively by ascertaining which replicas are operational when the replicated data object is first accessed and by sending this information along with the first write; the second write will contain the set of replicas which received the first write and so forth.

21 The original available copy protocol ..

The original available copy protocol [2, 71 relies on a complexmechanismto locatethat site. Severalsetsof failure informationare to be maintained in real time, including the set of sites participating in the replication of the data object and the sets of sites that had been specifidly included or

40 1

Similarly, when a replica t recovers from a replica s, s sends to t its new was-available set W, U { t } . Recovering from a total failurerequiresfinding the last site(s) that failed. These sites are known to belong to the closure of the wasavailable set with respect to the recovering site s, that is
C*(WS)=

Table 1. Example of failure and recovery

CA

I
I

CB
A,B,C A,B

U C"%)
k=O

1 1 1 1
kB
A.B.C We propose to record exact membership information in new metadata, which we will call cohort sets to distinguish them from was-available sets. A cohort set for any replica represents the set of replicas that participated in the last write that involved that replica. For example, if there are three replicas, A, B and C , and CA, CB and Cc are the corresponding cohort sets, Table 1 can be used to illustrate what happens when replicas fail and recover. Suppose that the system starts with a full complement of replicas. At some time in the future, replica C fails, and a write operation occurs. The state of the system is reflected in the second row of the table, where CA = CB = { A ,B } , which indicates that replicas A and B were the only participants in the last write operation. At some point further on, suppose that replica B also fails, followed by a write operation. This is reflected in the third row, where only replica A is current. Suppose now that the other two replicas rccover, then the state of the system is reflected in the fourth row. The only replica to be current is replica A because CA= { A } . Thus the rule is that, when k of the original n replicas have identical cohort sets all containing exactly these same k replicas and no others, we can assert that these k replicas are all current and all other replicas are stale. Hence version numbers are redundant. In general, we can also expect site failures to be much less frequent events than write operations. In this situation, cohort sets will almost always have been updated between consecutive site failures. Thus, after a failure of all sites holding replicas, the k current replicas, will also be the k last replicas that failed last. We can even expect k to be equal to one unless the last sites that failed did it so closely together that no write access took place during that time. The worst case is of course a simultaneous failure of all sites holding replicas. All replicas will be current but to establish this fact beyond any doubt we will need all sites holding to recover first. In this case, the new protocol will perform no better (and no worse) than the NAC protocol, which requires version numbers. In the following section, we considcr cohort sets a little more formally.

A,B,C A,B

I A,B,C

CC

3. A More Efficient Available Copy Protocol

One of the major objectives of the OAC protocol was to reduce the costs of updating Ihe was-available sets of operational sites [ 113. So it was decided that:
1. was-available set updates should always be piggy

backed on existing read, write and site recovery operations, and

2. was-available set updates should never involve sites that were not involved in each read, write or site recovery operation.
Hence, site recovery operations only updare the wasavailable set of the two sites actually participating in the actual recovery, namely the recovering site and that of the site from where the recovering site obtained the correct state of the replicated data object. As a result, the was-availablesets of the operational sites cease to be identical after a site recovery because only two operational sites will have included the site that recovered in their was-availablesets. Updating the was-availablesets of all operational sites would have had the two advantages of (a) making all available sets current and (b) removing the need to compute the closure of these sets every time the system has to recover from a total failure. As it happened, the OAC protocol was formalized [lo] well before its data availability was fully analyzed [ 11, 161. So the benefits of updating the was-availablesets at recovery time were only understood after the protocol had been fully specified and this update was done independentlyof the site recovery process itself [ 11, 161. An even more important simplificationcould be achieved if the was-availablesets could be alwaysbe correctly updated every time the replicated object is modified. We would know then that all the sites in the most recent was-available sets would all have the most recent version of the replicated object and would not need version nurnbers to distinguish them.

402

3.1. Cohort sets

A cohort set is the set of all replicas which have participated in the last write operation. The cohort sets are similar to the was-available sets used for the available copy protocol, except by managing them differently,we avoid the need to compute the closure. This results in a substantial simplificationin the recovery procedure.
Definition 1 A cohort set for a replica represents the set of replicas that were current during the last write that included this replica.

where every node has an edge to every other node. In this case, it is clear that having complete and equal cohort sets is equivalent to having a completely connected graph. If we now consider the write Operation, it will write identical cohort sets to all live: copies. These cohort sets list all of the live copies, and when viewed as a graph they form a completely connected subgraph. In our example, suppose that copy B fails, and this failure is followed by a write operation. Since B has Eailed, the write cannot change its cohort set, and so its out-going edges remain unchanged, but the edges from A and C to B are deletcd, and nodes A and C now form a completelyconnected subgraph. Node B is now a disconnected coimponent. Since no edges outside the completely connected subgraph are added, the subgraph remains unique. If we take the example one step further, suppose now that node C fails. Its edge to node A will remain, but there will be no edge back from node A and so node C is also in a disconnected component. Since node A has only the edge A 2 it is the unique completely connected subgraph. To understand the repair operation, consider Figure 2. The recovery operation is the inverse of the wrile operation. When a node which was dlisconnected is brought up to date, it is given the same cohort set as those nodes in the (unique) completely connected subgraph (which now includes the recovered node). Since these cohort sets do not include any nodes outside of the completely connected subgraph, the subgraph remains unique. Suppose now that node B is repaired. In this case, Cl, = CB = {A, B}, which, when % views as a graph means$iat node A has edges A 2 and A and node B has edges B B and B I . When node C recovers, it is handled in the exact same manner.

It should be clear that the last replica or replicas to fail must be represented by this set, since if any of the replicas had participated in subsequent write operation, then its cohort set wouldnot contain the replicas that didnot participate in this operation. It can be shown that the current set of replicas must have equal and complete cohort sets. By equal, we mean that the cohort sets for the replicas in question must have the same membership. By complete, we mean that every replica in question must be represented in the cohort sets, and every replica in the cohort sets must be present and under consideration.
Theorem 1 The necessary and sufJicient conditionfor recovery is that a subset of replicas can be found such that their cohort sets are equal and complete.

Instead of presenting a formal proof, which would be tedious and not very illustrative, we will demonstrate the result in an informal manncr. Perhaps the easiest way to understand this result, is to view it as a directed graph. Let each copy be represented by a node, and the cohort set of each copy represent the out-going edges from that node. That is, if copy A has C A = {A, B} then node A would have directed edges A> and A%.

I403

-p

Figure 2. A sequence of repair events.

32 Reading and writiing ..

The cohort sets are the sole mctadata requircd for accessing the replicated data. These sets must be consistent for the recovery algorithm to operate correctly. Hence they must be completely written to stable storage after the detection of a every failure. While extremely rare, it is possible that a second failure could oc(:ur while the cohort sets are being written.

Figure 1. A sequence of failure events.

If we consider Figure 1, we see that in the initial state, the system is represented by a completely connected graph,

There are several methods for insuring that the cohort sets are written to stable storage in a consistentfashion. The first is to leverage an existing commit mechanism [8]. The second is to employ a reliable multicast protocol [3]. We will describe a third method that uses a simple two phase write protocol to insure that the cohort sets are written in a consistent manner. In order to mitigate the effects of this unlikely failure scenario, two phases are used 10 write the cohort sets to stable storage, In the first phase, so-called tentative cohort sets are written. These sets are exactly like the regular cohort sets, but exist only briefly until the committed cohort sets have been safely written to disk. If this fails, the system can fall back to the original committed cohort sets. In the second phase, the tentative cohort sets are cleared and the committed cohort sets are written. Should this fail, then the tentative cohort sets that remain can be used in conjunction with the newly committed cohort sets to determine set of consistent replicas. The cohort sets are modified when a write operation occurs following a failure. It is assumed that write operations are frequent enough to provide sufficientlyfine grained failure detection. If this is not the case, then cohort sets can be modified when read operations occur. If an asynchronous failure notification mechanism is available, then this can be used to modify the cohort sets.

1. All cohort sets are tentative. This will succeed if there was a failure after the tentative cohort sets were written, but before the committed cohort sets were written.

2. A mix of tentative and committed cohort sets. This will succeed if there was a failurewhile the committed cohort sets were being written.
3. Only the committed cohort sets. This is the most common case, and will succeed if the two phase write completed successfully.

There is one further case, that is, when a failure occurs during the initial writing of the cohort sets. In this case,the cohort sets canbe safely ignored since the committedcohort sets represent a consistent view of the system.
The system considers each recovering replica in turn as it becomes active. This replica will compare its cohort set to the cohort sets of all other reovering replicas. When it is able to contact all replicas in its cohort set, and the cohort scts of each of these replicas agree with it ( h e equal and complete property), then this replica and all replicas in its cohort set can be declared to be current. Replicas which are unable to complete this procedure are out-of-dateand must repair from one of the current replicas, as discussed in Section 3.3.

3.3. Recovering individual replicas

In the absence of total failure, a recovering replica will find replicas that are available. It is then a simple matter to integrate this recovering replica into the st%of current replicas. First, the data from one of the current replicas is copied to the recovering replica. Second, the cohort set of one of the current replicas (recall that they are identical) is taken and the identity of the recovered replica is added to it. This new cohort set is then written to all current replicas, including the one which has just recovered.

4. Availability Analysis
Availability is the most common measure of fault tolerance for repairable systems that are expected to remain operational over a long period of time. It is wadi tionally defined as the fraction of time a system is operational. In the case of replicated data objects, the availabilityof a replicated object represents the fraction of time that the consistency control protocol will allow access to the object. The analysis of our new available copy protocol is identical to the analysis of the OAC protocol presented in [111 and [ 161. The system model consists of a set of sites with independent failure modes connected via a network which does not fail. When a site fails, a repair process is irmncdiately initiated at that site. Should several sites fail, the repair process will be performed in parallel on those sites. Site failures are assumed to be exponentially distributed with mean A, and repairs are aqsumed to be exponentiallydistributed with All mean ,U. access requests are assumed to be characterized by a Poisson process with mean K . The system is assumed to exist in statistical equilibrium. Although the assumption of

3.4. Recovering from a total failure

The recovery from total failure is the most intricate op-

eration in the system. As discussed in the previous section, the cohort sets must be carefully maintained. If we employ the two phase algorithm described in Section 3.2, then the following algorithm can be applied. In order to declare a set of replicas current, the cohort sets are checked for equality and completeness in the following order:

404

2h

Figure 3. State transition diagram

an independent failure rate X is reasonable if the sites have independentpower sources, the assumptions of exponential repair times and exponentialinter-access times are harder to defend on general grounds. However, all three assumptions are necessary to represent each system by a Markov process with a finite number of states [6].
Definition2 The availability of a replicated data object consisting of n replicas and managed by a replica control protocol S, denoted As ( n ) ,is the stationary probubility of the system being in a state where the replica controlprotocol will grant access to tlze data object.

Figure 4. Availability of two replicas managed by the new AC protocol

sets can be assumed to be up-to-date, then the availability is given by the expression:

The states of the Markov model are labeled by the ordered triple (i, j , k) where i represents the number of current (or up-to-date) replicas, j represents the cardinality of the current cohort set, and k represents the number of replicas that are out-of-date. When a triple is marked with a bar, for ex__ ample (1 , 2,0), this indicates that the system is unavailable. Figure 3 has the state transition diagram for two replicas managed by the new AC protocol. A system of equations cm be derived from this state transition diagram, and solved either algebraically or using numerical methods. If we let $ p = P and C = n, then the equations are significantly P simplified. The availability, A A C ( ~of ,the system is the sum of ) probabilities of being in a state where access is permitted, and is given by the expression:
AAC(2) = q5p2

The availability of a system with three replicas can be derived in a similar manner. In this case, the state diagram has sixteen states. The resulting expression is very large, and has been omitted for the sake of brevity. If we again make the assumptionof frequent writes, the then availability of system with three disks is given by the expression:

This analysis can be done for any number of replicas, though the equationsquickly becolme unmanageable. If the frequent write assumption is made, then a closed form solution has been derived [ll]. Figure 4 and 5 respectively represent the availability of two and three replicas managed by the new AC protocol for values of p varying between 0 and 0.25. We selected these values because a recent study [121 has shown that

+ 3p2 + 3 4 p +
(p

+ 1)3(p + 4 + 1)

4 p + q5

+1

If the writes occur with sufficient frequency h a t the cohort

405

partitions. A second applies to quorum-basedprotocols.

5.1. Protocols detecting data inconsistencies

Available copy protocols were designed for network topologies where network partitions were known to be impossible or extremely unlikely. Other protocols, such as the Coda replication control protocol [18, 141, follow the same write to aEUreud any philosophy as the available copy protocol but promise to detect ex post fucto data inconsistencies that may have resulted from these partitions. Detecting data inconsistencies in our protocol will involve comparing the cohort sets of the replicas searching for disjoint subsets of replicas such that every replica in each subset has a cohort set describing exactly that subset. If we find two or more of these subsets, there are two or more different versions of the file pretending to be the current version of the file. Otherwise we know that the data object has one single current state.
1

20 6 0.25

5.2. Extension to quorum-basedprotocols

Figure 5. Availability of three replicas managed by the new AC protocol

the mean time to failure (MTTF) for modem systems is approximately 29 days plus or minus 2. The mean time to repair (MTTR) is approximately 4 days plus or minus one. This results in reasonable values for y falling in the interval 0.06 < p < 0.22 for the average host connected to the Internet. Dedicated servers are likely to have service contracts which will result in a MlTR of one day or less, which will significantlylower the reasonable values of p into the range of 0.03 < p < 0.04. Professional maintenance and conditioned power will also significantly increase the MTTF, but these influences are more difficult to quantify. As one can see, the impact of the update rate to repair rate ratio 4 on the availability becomes insignificant as soon as q!~ > 4 or, in other words, K: > 4p.

Unlike availablecopy protocols, quorum-basedprotocols guarantee the consistency of the replicated data in the presence of network partitions. In their simplest form, quorumbased protocols assume that the correct state of a replicated object is the state of the majority of its replicas. Ascertaining Uie state of a replicated object requires collecting the votes of a quorunz of the replicas. Should this be prevented by a sufficient number of site failures, the replicated object is considered to be inaccessible. Protocols that adjusts quorums, such as dynamic voting and its variants [4,9], or modify the number of votes assigned to each replica [l], are known to provide higher data availability than protocol using static quorums. Cohort sets represent the set of replicas that participated in the last write operation. Whenever writes are significantly more frequent than site failures, they also provide a good approximationof the set of currently available replicas and can thus be used to implement dynamic voting protocols [131.

5. Possible Extensions

6. Conclusions
There are at least two possible extensions to our new protocol that are worth mentioning. The first extension concerns protocols that do not guarantee data consistency in presence of network partitions but promise to detect ex yosi fucto data inconsistencies that may have resulted from these Available copy protocols provide the highest data availability and data reliability of all replication protocols that do not regenerate failed replicas. Unfortunately,all existing impleinentations of available copy protocols either rely on

406

complex procedures for ascertaining which replicas are up to date after a total failure or have to wait for the recovery of all failed sites. We have presented a simple technique for efficiently implementing the available copy protocol. Our protocol does not require version numbers and maintains only n + log(n ) bits of state per replica, that is n bits for storing the current set of active replicas (the so-called cohort set and log(n) bits for storing the identity of the replica. The recovery procedure is also greatly simplified as it sufficesnow to gather all the replicas in any mutually agreed cohort set to find the current version of the replicated object. We have also shown that our new protocol provides the same data availability as the best feasible implementations of the available copy protocol. More work still needs to be done to extend the applicability of our technique and to investigate alternative implementations of the cohort set update process. One promising avenue would be to allow the cohort sets of some replicas to continue to include some replicas that failed before the last write but after the penultimate operation that recomputed the cohort set.

[5] D. K. Gifford, WeightedVoting for ReplicatedData, Proc. 7thACM Symposium on Operating System Principles, (1979), pp. 150-161. [61 B. V. Gnedenko, Mathematical Methods in Reliability Theory, Moscow, English Translation, New York, Academic Press, (1968). [7] N. Goodman, D. Slteen, A. Chan, U. Dayal, R. Fox and D. Ries, A Recovery Algorithm for a Distributed Database System, Proc. 2nd ACM Symposium on Principles of Datab,useSystems, (1983), pp. 8-15. [SI J. Gray and A. Reuter, Transaction Processing: Concepts and Techniques. Morgan Kaufman Publishers, San Mateo, Calif. (I 993). [9] S. Jajodia and D. Mutchler, Dynamic Voting Algorithms for Maintainingthe Consistency of a Replicated Database, ACM Tfiansactionson Database Systems, Vol. 15, No. 2 (1990), pp. 230-405. [ 101 D. D. E. Long and J.-.F.P%is, On Improving the Availability of Replicated Files, Proc. 6th Symposium on Reliable Distributea Systems, (1987), pp. 77-83. [111 D. D. E. Long, The Management of Replication in a Distributed System, Ph.D. dissertation, University of California, San Diego, 1988. [12] D. D.E. Long, A. IMuir, and R. Golding. A Longitudinal Study of Internet Host Reliability,Proc. 14th Symposium on Reliable Distributed Systems, (1999, pp. 2-9. [ 131 D. D. E. Long and J.-F. Piiris, Voting without Version Numbers, submitteld for publication. [14] L. B. Mummert, M. R. Ebling and M. Satyanarayanan, Exploiting Weak Connectivity for Mobile File Access: Proc. 15th ACM Symposium on Operating Systems Principles, (1993, pp. 33-45. [15] J.-F. Pkis, Voting with Witnesses: A Consistency Scheme for Replicated Files, Proc. 6th Znternational Conference on Distributed Computing Systems, (1986), pp. 606-612.. [161 J.-F. P%is and D. D. E. Long, On the Performance of Available Copy Protocols, Peflormance Evaluation, Vol. 11, (1990)pp. 9-30. [17] C. Pu, J. D. Noe and A. Proudfoot, Regenerationof Replicated Objects: A Technique and its Edcn Implementation, IEEE Transactionson Sojtware Engineering, Vol. SE-14, No. 7 (1988), pp. 936-945. [18] M. Satyanarayanan, J. J. Kistler, P Kumar, M. E. . Okasaki, E. H. Siegel, and D. C. Steere, Coda: A Highly Available File System for a Workstation Environment, IEEE Transactions on Computers, Vol. C-39, NO.4 (1990), pp. 447-459.

Acknowledgements
We are grateful for the input of our colleagues at the IBM Almaden Research Center, in particular L. F. Cabrera, N. Pass, N. Hanami, S. Edelman and A. Lam.

References
[l] D. Barbara, H. Garcia-Molina and A. Spauster, In-

creasing Availability Under Mutual Exclusion Constraints with Dynamic Vote Reassignment, ACM Transactions on Computer Systems, Vol. 7, No. 4 (1989), pp. 394-426. [2] P. A. Bernstein and N. Goodman, An algorithm for concurrency control and recovery in replicated distributed databases, ACM Transactions on Database System, Vol. 9, No. 4 (1984), pp. 596-615. [3] K. Birman and T. Joseph, Reliable Communication in the Presence of Failures, ACM Transactions on Computer Systems, Vol. 5, No.1 (1987), pp. 47-76. [4] D. Davkv and W. A. Burkhard, Consistency and Recovery Control for Replicated Files, Proc. 10th ACM

Symposium on Operating System Principles, (1985) pp. 87-96.

407