Advanced Operating Systems (Distributed Systems)

Unit 2

Unit 2
Structure: 2.1 Introduction Objectives 2.2 2.3 2.4 2.5 2.6 2.7 2.8 2.9 Features of Message Passing Issues in IPC by Message Passing Synchronization Buffering Process Addressing Failure Handling Group Communication Terminal Questions

Message Passing

2.1 Introduction
A process is a program in execution. When we say that two computers of a distributed system are communicating with each other, we mean that two processes, one running on each computer, are in communication with each other. A distributed operating system needs to provide interprocess communication (IPC) mechanisms to facilitate such communication activities. A message passing system is a subsystem of the distributed operating system which shields the details of complex network protocols from the programmer. It enables processes to communicate by exchanging messages and allows programs to be written by using simple

communication primitives such as send and receive. Interprocess communication basically requires information sharing among two or more processes. The two basic methods for information sharing are as follows:

Sikkim Manipal University

Page No. 43

Advanced Operating Systems (Distributed Systems)

Unit 2

i)

Original Sharing or Shared Data approach Message is placed in a common memory area that is accessible to all processes. This is not possible in a distributed system, unless it is a distributed shared memory system (DSM).

ii) Copy Sharing or Message Passing approach Message is physically copied from senders address space to the receivers address space. This is the basic IPC mechanism in distributed systems.

In the shared data approach, the information to be shared is placed in a common memory area that is accessible to all the processes involved in an IPC. The shared data paradigm gives the conceptual communication pattern illustrated in figure 2.1 below:
Shared Common Memory Area

P1

P2

Pn-1

Pn

Figure 2.1: Communications in Shared Data Paradigm

In the method of message passing, the information to be shared is physically copied from the sender processs address space to the address space of all the receiver processes, and this is done by transmitting the data to be copied in the form of messages (a message is a block of information). The message passing paradigm gives the conceptual communication
Sikkim Manipal University Page No. 44

Advanced Operating Systems (Distributed Systems)

Unit 2

pattern as shown in figure 2.2 below. In this case the communicating processes interact directly with each other.
P1 P2

Figure 2.2: Communication in Message Passing Paradigm

Since computers in a network do not share memory, processes in a distributed system normally communicate by exchanging messages with among them. Therefore message passing is the basic IPC mechanism in distributed systems.

2.2 Features of a Message Passing System

Desirable features of a good message passing system are: Simplicity Efficiency Reliability Correctness Flexibility Security Portability Simplicity The message passing system should be easy to use easy to develop new applications that communicate with the existing ones able to hide the details of underlying network protocols used

Efficiency Should reduce the number of message exchanges (acks,..)

Page No. 45

Sikkim Manipal University

Advanced Operating Systems (Distributed Systems)

Unit 2

Avoid the costs of establishing and terminating connections between the same pair of processes for each and every message Piggyback acknowledgments with the normal messages Send acknowledgments selectively

Reliability Should handle node and link failures Normally handled by acknowledgments, timeouts and

retransmissions. Should handle duplicate messages that arise due to retransmissions (generally sequence numbers of the messages are used for this purpose). Correctness Atomicity: messages sent to a group of processes will be delivered to all of them or none of them. Ordered delivery: Messages are received by all receivers in an order acceptable to the application. Survivability: Guarantees messages will be delivered correctly in spite of failures. Flexibility IPC protocols should be flexible to cater to the various needs different applications (i.e. some may not require atomicity others may not require ordered delivery, etc) IPC primitives should be flexible to permit any kind of control flow between cooperating processes, including synchronous and

asynchronous send and receive. Security Message passing system should be capable of providing secure end-to-end communication.
Sikkim Manipal University Page No. 46

Advanced Operating Systems (Distributed Systems)

Unit 2

Support mechanisms for authentication of the receivers of a message by a sender. Support mechanisms for authentication of the sender by its receivers Support encryption of a message before sending it over the network.

Portability: There are two different aspects of portability in a messagepassing system: 1. The message-passing system should itself be portable. It should be possible to easily construct a new IPC facility on another system by reusing the basic design of the existing message-passing system. 2. The applications written by using the primitives of the IPC protocols of the message-passing system should be made portable. This requires that heterogeneity must be considered while designing a message passing system. This may require the use of external data representation format for the communication taking place between two or more processes running on computers of different architectures.

2.3 Issues in IPC (Inter-process Communication) by Message Passing

A message is a meaningful formatted block of information sent by the sender process to the receiver process. The message block consists of a fixed length header followed by a variable size collection of typed data objects. The header block of a message may have the following elements: Address: A set of characters that uniquely identify both the sender and receiver. Sequence Number: It is the Message Identifier to identify duplicate and lost messages in case of system failures.
Sikkim Manipal University Page No. 47

Advanced Operating Systems (Distributed Systems)

Unit 2

Structural Information: It has two parts. The type part that specifies whether the data to be sent to the receiver is included within the message or the message only contains a pointer to the data. The second part specifies length of the variable-size message.

In a message oriented IPC protocol, the users are fully aware of the message formats used in the communication process and the mechanisms used to send and receive messages. The following are some important issues to be considered for the design of an IPC protocol based message passing system: The Senders Identity The Receivers Identity Number of Receivers Guaranteed acceptance of sent messages by the receiver Acknowledgement by the sender Handling system crashes or link failures Handling of buffers Order of delivery of messages

The above issues are addressed by the semantics of the communication primitives provided by the IPC Protocol. A general descriptionof the various ways in which these issues are addressed by message oriented IPC protocols is presented below.

2.4 Synchronization
A major issue in communication is the synchronization imposed on the communicating processes by the communication primitives. There are two types of communicating primitives: Blocking Semantics and Non-Blocking Semantics.

Sikkim Manipal University

Page No. 48

Advanced Operating Systems (Distributed Systems)

Unit 2

Blocking Semantics: A communication primitive is said to have blocking semantics if its invocation blocks the execution of its invoker (for example in the case of send, the sender blocks until it receives an acknowledgement from the receiver.)

Non-blocking Semantics: A communication primitive is said to have non-blocking semantics if its invocation does not block the execution of its invoker.

The synchronization imposed on the communicating processes basically depends on one of the two types of semantics used for the send and receive primitives. Blocking Primitives Blocking Send Primitive: In this case, after execution of the send statement, the sending process is blocked until it receives an

acknowledgement from the receiver that the message has been received. Non-Blocking Send Primitive: In this case, after execution of the send statement, the sending process is allowed to proceed with its execution as soon as the message is copied to the buffer. Blocking Receive Primitive: In this case, after execution of the receive statement, the receiving process is blocked until it receives a message. Non-Blocking Receive Primitive: In this case, the receiving process proceeds with its execution after the execution of receive statement, which returns the control almost immediately just after telling the kernel where the message buffer is. Handling non-blocking receives: The following are the two ways of doing this: Polling: a test primitive is used by the receiver to check the buffer status

Sikkim Manipal University

Page No. 49

Advanced Operating Systems (Distributed Systems)

Unit 2

Interrupt: When a message is filled in the buffer, software interrupt is used to notify the receiver. However, user level interrupts make programming difficult.

Handling blocking receives: A timeout value may be used with a blocking receive primitive to prevent a receiving process from getting blocked indefinitely if the sender has failed. Synchronous Vs Asynchronous Communication When both send and receive primitives of a communication between two processes use blocking semantics, the communication is said to be synchronous. If one or both of the primitives is non-blocking, then the communication is said to be asynchronous. Synchronous communication is easy to implement. It contributes to the reliable delivery of messages. Asynchronous communication limits

concurrency and is prone to communication deadlocks.

2.5 Buffering
The transmission of messages from one process to another can be done by copying the body of the message from the senders address space to the receivers address space. In some cases, the receiving process may not be ready to receive the message but it wants the operating system to save that message for later reception. In such cases, the operating system would rely on the receivers buffer space in which the transmitted messages can be stored prior to receiving process executing specific code to receive the message. The synchronous and asynchronous modes of communication correspond to the two extremes of buffering: a null buffer, or no buffering, and a buffer with unbounded capacity. Two other commonly used buffering strategies are

Sikkim Manipal University

Page No. 50

Advanced Operating Systems (Distributed Systems)

Unit 2

single-message and finite-bound, or multiple message buffers. These four types of buffering strategies are given below: No buffering: In this case, message remains in the senders address space until the receiver executes the corresponding receive. Single message buffer: A buffer to hold a single message at the receiver side is used. It is used for implementing synchronous communication because in this case an application can have only one outstanding message at any given time. Unbounded - Capacity buffer: Convenient to support asynchronous communication. However, it is impossible to support unbounded buffer. Finite-Bound communication. Buffer overflow can be handled in one of the following ways: Unsuccessful communication: send returns an error message to the sending process, indicating that the message could not be delivered to the receiver because the buffer is full. Flow-controlled communication: The sender is blocked until the receiver accepts some messages. This violates the semantics of asynchronous send. This will also result in communication deadlock. A message data should be meaningful to the receiving process. This implies ideally that the structure of the program should be preserved while they are being transmitted from the address space of the sending process to the address space of the receiving process. It is not possible in heterogeneous systems in which the sending and receiving processes are on computers of different architectures. Even in homogeneous systems, it is very difficult to achieve this goal mainly because of two reasons: Buffer: Used for supporting asynchronous

Sikkim Manipal University

Page No. 51

Advanced Operating Systems (Distributed Systems)

Unit 2

1. An absolute pointer value has no meaning (more on this when we talk about RPC). For example, a pointer to a tree or linked list. So, proper encoding mechanisms should be adopted to pass such objects. 2. Different program objects, such as integers, long integers, short integers, and character strings occupy different storage space. So, from the encoding of these objects, the receiver should be able to identify the type and size of the objects. One of the following two representations may be used for the encoding and decoding of a message data: 1. Tagged representation: The type of each program object as well as its value is encoded in the message. In this method, it is a simple matter for the receiving process to check the type of each program object in the message because of the self-describing nature of the coded data format. 2. Untagged representation: The message contains only program objects, no information is included in the message about the type of each program object. In this method, the receiving object should have a prior knowledge of how to decode the received data because the coded data format is not self-describing. The untagged representations used in SUNs XDR format and tagged representation is used in Mach distributed operating system.

2.6 Process Addressing

A message passing system generally supports two types of addressing: Explicit Addressing: The process with which communication is desired is explicitly specified as a parameter in the communication primitive. e.g. send (pid, msg), receive (pid, msg).

Sikkim Manipal University

Page No. 52

Advanced Operating Systems (Distributed Systems)

Unit 2

Implicit Addressing: A process does not explicitly name a process for communication. For example, a process can specify a service instead of a process. e.g. send any (service id, msg), receive any (pid, msg) Methods for process addressing: machine id@local id: UNIX uses this form of addressing (IP address, port number). Advantages: No global coordination needed for process addressing. Disadvantages: Does not allow process migration. machine id1@local id@machine id2: machine id1 identifies the node on which the process is created. local id is generated by the node on which the process is created. machine id2 identifies the last known location of the process. When a process migrates to another node, the link information (the machine id to which the process migrates) is left with the current machine. This information is used for forwarding messages to migrated processes. Disadvantages: Overhead involved in locating a process may be large. If the node on which the process was executing is down, it may not be possible to locate the process.

2.7 Failure Handling

While a distributed system may offer potential for parallelism, it is also prone to partial failures such as a node crash or a communication link failure. During Interprocess communication, such failures may lead to the following problems: Loss of request Message: This can be due to link failure or receiver node is down.

Sikkim Manipal University

Page No. 53

Advanced Operating Systems (Distributed Systems)

Unit 2

Loss of response message: This may be due to link failure or the sender is down when the response reaches it. Unsuccessful execution of request: This may be due to receiver node crash while processing the request.

A Solution to overcome the above said problem may be done using the following methods: 1. A four message reliable IPC: In this method there are four messages involved: Request and Acknowledgement from the client machine, Reply and Acknowledgement from the Server machine. In this case, the kernels of both the client and server will continue to retransmit after timeout until an acknowledgement is received from both, i.e the client machine sends a request message to the server machine and waits for an acknowledgement from the server. If the acknowledgement is not received within the specified timeout period, the client retransmits its request to the server and waits for an

acknowledgement. This process continues till an acknowledgement is received. The same process occurs even at the server side. The server sends a reply message to the client and waits for the acknowledgement until the specified timeout period. On non-receipt of the acknowledgement within the timeout period, it resends the reply back to the client machine and the process continues till the client responds with an acknowledgement. 2. Three message reliable IPC: As mentioned in point number 1 above, the scenario here slightly varies, wherein the client machine does not wait for an acknowledgement to be received from the server machine. The client machine just sends the request to the specified server. But here the server machine expects an acknowledgement from the client machine when it responds to the clients
Sikkim Manipal University Page No. 54

Advanced Operating Systems (Distributed Systems)

Unit 2

request message. The server now waits for an acknowledgement from the client and on non-receipt of the acknowledgement within the specified time period, it retransmits the reply message to the client and this cycle continues until the client responds with an acknowledgement. In this method, the server may use the concept of piggybacking, wherein it may attach the acknowledgement to the client with a message in the form of a reply to the client. 3. Two message reliable IPC: In this method there is no requirement either from the client or the server for receiving acknowledgements from each other. They just exchange the messages in the form of requests and replies or responses to each other assuming that their messages have been sent (ideal scenario), but which may be impractical in real time situations. Idempotency and handling of duplicate request messages Idempotency basically means repeatability. i.e. an Idempotent operation produces the same results without any side effects no matter how many times it is performed with the same arguments. For example, assume an sqrt procedure for calculating the square root of a given number; sqrt (64) always returns 8. On the other hand, operations that do not necessarily produce the same results when executed repeatedly with the same arguments are said to be non-idempotent. For example a debit operation on a bank account. An idempotent operation produces the same result without any side effect no matter how many times it is executed. Not all operations are idempotent So, if requests can be retransmitted, then care should be taken to implement its reply as an idempotent operation.

Sikkim Manipal University

Page No. 55

Advanced Operating Systems (Distributed Systems)

Unit 2

Even if the same request is retransmitted several times, the server should execute the request only once; or if it executes several times, the net result should be equivalent to the result of exactly one execution. This is called exactly once semantics. Primitives based on exactly-once semantics are desirable but difficult to implement.

Implementation of exactly-once semantics: each request has a unique sequence number Kernel makes sure request is forwarded to server only once After receiving the reply from the server, Kernel caches a copy of the reply and retransmits it when it receives the same request from client

2.8 Group Communication

The most elementary form of message-based interaction is one-to-one communication in which a single-sender process sends a message to a single receiver process. However, for performance and ease of

programming several highly parallel distributed applications require that a message passing system should also provide group communication facility. Depending on single or multiple senders and receivers, the following three types of group communication are possible: 1. One-to-many (Single sender and multiple receivers) 2. Many-to-one (multiple senders and single receiver) 3. Many-to-many (multiple senders and multiple receivers) The following are the One-to-Many Multicast issues in a group communication to be addressed: i) Group Management: In case of one-to-many communication, receiver processes of a message form a group. Such groups are of two types closed and open. A closed group is one in which only the members of the group can send a message
Sikkim Manipal University Page No. 56

Advanced Operating Systems (Distributed Systems)

Unit 2

to the group. An outside process cannot send a message to the group as a whole, although it may send a message to an individual member of the group. On the other hand an open group is one in which any process in the system can send a message to the group as a whole. Whether to use a closed group or an open group is application dependent. A message passing system with group communication facility provides the flexibility to create an delete groups dynamically and to allow a process to join or leave a group at any time. ii) Group Addressing: A two-level naming scheme is normally used for group addressing. The higher level group is an ASCII string that is independent of the location information of the processes in the group. The low-level group name depends to a large extent on the underlying hardware. For example, on some networks it is possible to create a special network address to which multiple machines can listen. Create a special network address, called multicast address. A packet sent to multicast address is delivered to all who have subscribed to that group. For example, on the Internet, class D IP addresses are used for multicast. The format of class D IP addresses for IP multicasting: -------------------------------------|1|1|1|0| Group identification| -------------------------------------The first four bits contain 1110 and identify the address as multicast. The remaining 28 bits specify a specific multicast group. Broadcast address: A certain address is declared as a broadcast address and packets sent to that address are delivered to all in the network.
Sikkim Manipal University Page No. 57

Advanced Operating Systems (Distributed Systems)

Unit 2

If there is no facility to create multicast or broadcast addresses, then underlying unicast is used. A disadvantage is for each member a separate copy of each packet needs to be sent.

iii) Message Delivery Approach: The following are the two possible approaches for message delivery. Centralized approach: A centralized group server maintains

information about the groups and their members. Decentralized approach: No central server keeps the information.

Buffered or Unbuffered: A multicast packet can be buffered until the receiver is ready to receive. If unbuffered, packets could be lost. Multicast send is inherently asynchronous: It is unrealistic to expect sending process to wait until all the receiving processes that belong to the multicast group are ready to receive. The sending process may not be aware of all the receiving processes

Flexible Reliability in Multicast Communication: Different levels of reliability O-reliable: No response is expected from any receivers. 1-reliable: Sender expects response from one receiver (may be the multicast server can take the responsibility). m-out-of-n-reliable: The sender expects response from m out of n receivers. All-reliable: The sender expects response from all receivers. Atomic Multicast: A multicast message is received by all the members of the group or none. Different Implementation methods: The Kernel of the sender is responsible for retransmitting until everyone receives. This method works only if the senders machine and none of the receiver processes fail.
Sikkim Manipal University Page No. 58

Advanced Operating Systems (Distributed Systems)

Unit 2

Each receiver of the multicast message performs an atomic multicast of the same message to the same group. This method ensures all surviving processes will receive the message even if some receivers fail after receiving the message or the sender machine fails after sending the message.

iv) Many-to-One Communication: In this type of communication, multiple senders send messages to a single receiver. For example, A buffer process may receive messages from several consumers and producers. Multicast recipients may be sending acknowledgements to the sender. A database server may be receiving requests from several clients

v) Many-to-Many Communication: In this type of communication, multiple senders send messages to multiple receivers. An important issue here is that of ordered delivery of messages. Ordered delivery ensures that all messages are delivered to all receivers in an order acceptable to the application. The following are the various message ordering semantics followed in case of a Many-to-Many communication: i) Absolute Ordering: In this type, all messages are delivered to all processes in the exact order in which they were sent. Not possible to implement in the absence of global clock. Moreover, absolute ordering is not required by many applications. ii) Consistent Ordering: In this type, all messages are received by all processes in the same order. iii) Causal Ordering: For some applications, consistent-ordering

semantics is not necessary and even weaker semantics is acceptable. An application can have better performance if the message-passing
Sikkim Manipal University Page No. 59

Advanced Operating Systems (Distributed Systems)

Unit 2

system used supports a weaker ordering semantics that is acceptable to the application. One such weak ordering semantics that is acceptable to many applications is the causal ordering semantics. This semantics ensures that if the event of sending one message is causally related to the event of sending another message, the two messages are delivered to all receivers in the correct order. Two message sending events are said to be causally related if they are correlated by the happened-before relation. i.e. two message sending events are causally related if there is any possibility of the second one being influenced in any way by the first one. The basic idea behind causal ordering semantics is that when it matters, messages are always delivered in proper order, but when it does not matter, they may be delivered in any arbitrary order.

2.9 Terminal Questions

1. What is a message passing system? Discuss the desirable features of a message passing system. 2. Discuss the synchronization issues in a message passing system. 3. Discuss the issues of buffering and process addressing. 4. Discuss about group communication mechanisms in Message Passing.

Sikkim Manipal University

Page No. 60