Motivation for Distribution

Concurrent Object-Oriented
Network Programming with C++  Benets of distributed computing:
{ Collaboration ! connectivity and interworking
{ Performance ! multi-processing and locality
Douglas C. Schmidt { Reliability and availability ! replication
Washington University, St. Louis { Scalability and portability ! modularity
{ Extensibility ! dynamic conguration and recon-
http://www.cs.wustl.edu/schmidt/ guration
schmidt@cs.wustl.edu { Cost eectiveness ! open systems and resource
sharing

1 2

Challenges and Solutions Caveats

 OO and C++ are not a panacea
 Developing ecient, robust, and extensible
distributed applications is challenging { However, when used properly they help minimize
\accidental" complexity and improve software qual-
{ e.g., must address complex topics that are less ity
problematic or not relevant for non-distributed and
non-concurrent applications
 Advanced OS features provide additional func-
tionality and performance, e.g.,
 Object-oriented (OO) techniques and C++
language features enhance distributed soft- { Multi-threading
ware quality factors
{ Key OO techniques ! design patterns and frame- { Multi-processing
works
{ Synchronization
{ Key C++ language features ! classes, inheritance,
dynamic binding, and parameterized types { Explicit dynamic linking
{ Key software quality factors ! modularity, exten- { Shared memory
sibility, portability, reusability, reliability, and cor-
rectness
{ Communication protocols and IPC mechanisms

3 4
 Tutorial Outline Software Development
Environment
 Outline key challenges for developing dis-
tributed applications  The topics discussed here are largely inde-
pendent of OS, network, and programming
language
 Present an OO design and implementation { Currently being used on UNIX and Windows NT
of the following communication applications: platforms, running on TCP/IP and IPX/SPX net-
1. Distributed Logger works, written in C++

2. Application-level Gateway
 Examples illustrated with freely available ADAP-
TIVE Communication Environment (ACE)
OO toolkit
 Both single-threaded and multi-threaded so- { Although ACE is written in C++, the principles
lutions are given apply to other OO languages

5 6

Stand-alone vs. Distributed

Application Architectures Sources of Complexity
PRINTER

COMPUTER  Distributed application development exhibits

FILE
both inherent and accidental complexity
CD ROM SYSTEM

 Inherent complexity results from fundamen-

(1) STAND-ALONE APPLICATION ARCHITECTURE
tal challenges in the distributed application
NAME
TIME
domain, e.g.,
SERVICE CYCLE

DISPLAY
SERVICE SERVICE
{ Addressing the impact of latency
SERVICE
{ Detecting and recovering from partial failures of
networks and hosts
FI LE
NETWORK SERVICE

PRINT
SERVICE { Load balancing and service partitioning
CD ROM

PRINTER FILE SYSTEM { Consistent ordering of distributed events

(2) DISTRIBUTED APPLICATION ARCHITECTURE

7 8
 OO Contributions to Distributed
Application Development
Sources of Complexity (cont'd)
 Distributed application implementors tradi-
tionally used low-level IPC and OS mecha-
 Accidental complexity results from limita- nisms, e.g.,
tions with tools and techniques used to de- { sockets ! I/O descriptors, address formats, byte-
velop distributed applications, e.g., ordering, signals, message buering and framing
{ Lack of type-safe, portable, re-entrant, and exten- { select ! bitmasks, signals, descriptor counts, time-
sible system call interfaces and component libraries
outs
{ Inadequate debugging support { fork/exec ! signal handling, I/O descriptors
{ Widespread use of algorithmic decomposition { POSIX pthreads and Solaris threads
. Fine for explaining network programming con-
cepts and algorithms but inadequate for devel-  OO design patterns and frameworks elevate
oping large-scale distributed applications focus onto on application requirements and
policies, e.g.,
{ Continuous rediscovery and reinvention of core con-
cepts and components { Service functionality
{ Service conguration
{ Service concurrency
9 10

Example Distributed Application Distributed Logging Service

PRINTER  Server logging daemon
P1
CLIENT
CONSOLE
{ Collects, formats, and outputs logging records for-
P2
LOCAL IPC LOGGING
warded from client logging daemons residing through-
P3
DAEMON
RE
MO
TE
out a network or internetwork
IPC
SERVER LOGGING
DAEMON
HA
OST A B B
HOST

HOST
A CLIENT SERVER  The application interface is similar to printf
ACE_ERROR ((LM_ERROR, "(%t) can't fork in function spawn"));
IPC
TE
MO

// generates on server host

NETWORK
RE

STORAGE
DEVICE
Oct 29 14:50:13 1992@crimee.ics.uci.edu@22766@7@client
HOST P1 ::(4) can't fork in function spawn
CLIENT
B CLIENT
LOCAL IPC LOGGING
P2 DAEMON
ACE_DEBUG ((LM_DEBUG,
P3 "(%t) sending to server %s", server_host));

// generates on server host

 Distributed logging service Oct 29 14:50:28 1992@zola.ics.uci.edu@18352@2@drwho

::(6) sending to server bastille

11 12
 Conventional Logging Server
Design Select-based Logging Server
Implementation
 Typical algorithmic pseudo-code for the server
daemon portion of the distributed logging SERVER
service: SERVER
LOGGING DAEMON

void server logging daemon (void) acceptor

maxhandlep1
f NETWORK read_handles
initialize listener endpoint
loop forever CONNECTION
f REQUEST
wait for events CLIENT

handle data events

handle connection events LOGGING
g RECORDS LOGGING LOGGING
g RECORDS RECORDS

 The \grand mistake": CLIENT

CLIENT CLIENT

{ Avoid the temptation to \step-wise rene" this

algorithmically decomposed pseudo-code directly
into the detailed design and implementation of the
logging server!
13 14

Conventional Logging Server

Implementation
// Run main event loop of server logging daemon.

 Note the excessive amount of detail required
to program at the socket level :::
// Main program
static const int PORT = 10000;
int main (int argc, char *argv[])
{
initialize_listener_endpoint
(argc > 1 ? atoi (argv[1]) : PORT);
// Loop forever performing logging server processing.

typedef u_long COUNTER; for (;;) {

typedef int HANDLE; tmp_handles = read_handles; // struct assignment.

// Counts the # of logging records processed // Wait for client I/O events
static COUNTER request_count; select (maxhp1, &tmp_handles, 0, 0, 0);

// Passive-mode socket descriptor // First receive pending logging records

static HANDLE listener; handle_data ();

// Highest active descriptor number, plus 1 // Then accept pending connections

static HANDLE maxhp1; handle_connections ();
}
// Set of currently active descriptors }
static fd_set read_handles;

// Scratch copy of read_handles

static fd_set tmp_handles;

15 16
 // Receive pending logging records
// Initialize the passive-mode socket descriptor
static void handle_data (void)
static void initialize_listener_endpoint (u_short port) {
{ // listener + 1 is the lowest client descriptor
struct sockaddr_in saddr;
for (HANDLE h = listener + 1; h < maxhp1; h++)
// Create a local endpoint of communication if (FD_ISSET (h, &tmp_handles)) {
listener = socket (PF_INET, SOCK_STREAM, 0); ssize_t n;

// Set up the address information to become a server
memset ((void *) &saddr, 0, sizeof saddr);
saddr.sin_family = AF_INET;
saddr.sin_port = htons (port);
saddr.sin_addr.s_addr = htonl (INADDR_ANY);
// Associate address with endpoint
bind (listener, (struct sockaddr *) &saddr, sizeof saddr);
// Make endpoint listen for connection requests
listen (listener, 5);
// Guaranteed not to block in this case!
if ((n = handle_log_record (h, 1)) > 0)
++request_count; // Count the # of logging records
else if (n == 0) { // Handle connection shutdown.
FD_CLR (h, &read_handles);
close (h);
if (h + 1 == maxhp1) {
// Skip past unused descriptors

// Initialize descriptor sets while (!FD_ISSET (--h, &read_handles))

FD_ZERO (&tmp_handles); continue;
FD_ZERO (&read_handles);
FD_SET (listener, &read_handles); maxhp1 = h + 1;
}
maxhp1 = listener + 1; }
} }
}

17 18

// Receive and process logging records

static ssize_t handle_log_record

(HANDLE in_h, HANDLE out_h)
{ // Check if any connection requests have arrived
ssize_t n;
size_t len; static void handle_connections (void)
Log_Record log_record; {
if (FD_ISSET (listener, &tmp_handles)) {
// The first recv reads the length (stored as a static struct timeval poll_tv = {0, 0};
// fixed-size integer) of adjacent logging record. HANDLE h;

n = recv (in_h, (char *) &len, sizeof len, 0); // Handle all pending connection requests
// (note use of select's "polling" feature)
if (n <= 0) return n;
do {
len = ntohl (len); // Convert byte-ordering h = accept (listener, 0, 0);
FD_SET (h, &read_handles);
// The second recv then reads LEN bytes to obtain the
// actual record // Grow max. socket descriptor if necessary.
for (size_t nread = 0; nread < len; nread += n if (h >= maxhp1)
n = recv (in_h, ((char *) &log_record) + nread, maxhp1 = h + 1;
len - nread, 0); } while (select (listener + 1, &tmp_handles,
0, 0, &poll_tv) == 1);
// Decode and print record. }
decode_log_record (&log_record);
write (out_h, log_record.buf, log_record.size);
return n;
}

19 20
 Limitations with Algorithmic
Decomposition Techniques Overcoming Limitations via OO
 Algorithmic decomposition tightly couples  The algorithmic decomposition illustrated
application-specic functionality and the fol- above species many low-level details
lowing conguration-related characteristics:
{ Structure { Furthermore, the excessive coupling signicantly
complicates reusability, extensibility, and portability: : :
. The number of services per process
. Time when services are congured into a process  In contrast, OO focuses on application-specic
{ Communication Mechanisms behavior, e.g.,
. The underlying IPC mechanisms that communi- int Logging_Handler::svc (void)
cate with other participating clients and servers {
ssize_t n;

. Event demultiplexing and event handler dispatch- n = handle_log_record (peer ().get_handle (), 1);
ing mechanisms if (n > 0)
++request_count; // Count the # of logging records
{ Execution Agents }
return n <= 0 ? -1 : 0;

. The process and/or thread architecture that ex-

ecutes service(s) at run-time
21 22

The ADAPTIVE Communication

OO Contributions Environment (ACE)
OO application frameworks achieve large-
DISTRIBUTED

 SERVICES AND GATEWAY TOKEN LOGGING NAME TIME

scale design and code reuse COMPONENTS SERVER SERVER SERVER SERVER SERVER

{ Frameworks emphasize the collaboration among

FRAMEWORKS SERVICE CORBA
ACCEPTOR CONNECTOR
AND CLASS HANDLER HANDLER

societies of classes and objects in a domain CATEGORIES

ADAPTIVE SERVICE EXECUTIVE (ASX)

THREAD LOG SERVICE SHARED

C++

{ In contrast, traditional techniques focus on the

MANAGER MSG CONFIG- MALLOC
WRAPPERS
REACTOR URATOR

functions and algorithms that solve particular re-

SYNCH SPIPE SOCK_SAP/ FIFO MEM SYSV
WRAPPERS SAP TLI_SAP SAP MAP WRAPPERS

quirements C THREAD STREAM SOCKETS/

OS ADAPTATION LAYER
NAMED SELECT/ DYNAMIC MEMORY SYSTEM
APIS LIBRARY PIPES TLI PIPES POLL LINKING MAPPING V IPC
PROCESS/THREAD COMMUNICATION VIRTUAL MEMORY
SUBSYSTEM SUBSYSTEM SUBSYSTEM

 OO design patterns facilitate the large-scale GENERAL POSIX AND WIN32 SERVICES

reuse of software architecture

{ Even when reuse of algorithms, detailed designs,
and implementations is not feasible
 A set of C++ wrappers, class categories,
and frameworks based on design patterns

23 24
 Class Categories in ACE (cont'd)
Class Categories in ACE  Responsibilities of each class category
{ IPC encapsulates local and/or remote IPC mech-
APPLICATION- APPLICATIONS
APPLICATIONS
anisms
SPECIFIC APPLICATIONS
{ Service Initialization encapsulates active/passive
connection establishment mechanisms
Network
Services { Concurrency encapsulates and extends multi-threading
Stream
APPLICATION- Framework and synchronization mechanisms
INDEPENDENT
Service { Reactor performs event demultiplexing and event
Initialization handler dispatching
Interprocess { Service Configurator automates conguration
Communication Service and reconguration by encapsulating explicit dy-
Configurator namic linking mechanisms
Reactor { Stream Framework models and implements layers
Concurrency
global and partitions of hierarchically-integrated commu-
nication software
{ Network Services provides distributed naming,
logging, locking, and routing services
25 26

Graphical Notation
Design Patterns
 Design patterns represent solutions to prob-
OBJECT lems that arise when developing software
PROCESS : CLASS within a particular context
THREAD
{ i.e., \Patterns == problem/solution pairs in a con-
text"
CLASS
CLASS TEMPLATE UTILITY
CLASS  Patterns capture the static and dynamic struc-
ture and collaboration among key partici-
pants in software designs
CLASS
INHERITS INSTANTIATES { They are particularly useful for articulating how
CATEGORY
and why to resolve non-functional forces

ABSTRACT
CLASS
A
CONTAINS USES  Patterns facilitate reuse of successful soft-
ware architectures and designs

27 28
 Design Patterns in the
Distributed Logger Design Patterns in the
Distributed Logger (cont'd)
Active
Object  Reactor pattern
Service { Decouple event demultiplexing and event handler
Acceptor Configurator dispatching from application services performed in
response to events
STRATEGIC
Reactor
PATTERNS
 Acceptor pattern
TACTICAL Template Factory { Decouple the passive initialization of a service from
PATTERNS Iterator Method Method Adapter the tasks performed once the service is initialized

29 30

OO Logging Server
 The object-oriented server logging daemon
is decomposed into several modular compo-
nents that perform well-dened tasks:
Design Patterns in the 1. Application-specic components
Distributed Logger (cont'd) { Process logging records received from clients
 Service Congurator pattern 2. Connection-oriented application components
{ Decouple the behavior of network services from { Svc Handler
point in time at which services are congured into
an application . Performs I/O-related tasks with connected clients
{ Acceptor factory
 Active Object pattern . Passively accepts connection requests from clients

{ Decouple method invocation from method execu- . Dynamically creates a Svc Handler object for
tion and simplies synchronized access to shared each client and \activates" it
resources by concurrent threads
3. Application-independent ACE framework compo-
nents
{ Perform IPC, explicit dynamic linking, event de-
multiplexing, event handler dispatching, multi-
threading, etc.
31 32
 Class Diagram for OO Logging
Server
Logging_Handler SOCK_Stream
The Reactor Pattern
APPLICATION-

COMPONENTS

SOCK_Acceptor Null_Synch
1 n
SPECIFIC

Logging
Acceptor
Logging  Intent
ACTIVATES Handler
{ \Decouple event demultiplexing and event han-
dler dispatching from the services performed in
response to events"
CONNECTION-

COMPONENTS

SVC_HANDLER PEER_STREAM
ORIENTED

PEER_ACCEPTOR SYNCH
Svc
Acceptor Handler  This pattern resolves the following forces
for event-driven software:
{ How to demultiplex multiple types of events from
PEER
ACCEPTOR
PEER
multiple sources of events eciently within a single
thread of control
STREAM
COMPONENTS
FRAMEWORK

Stream
Connection
{ How to extend application behavior without requir-
ACE

IPC_SAP ing changes to the event dispatching framework

Service
Configurator

Concurrency Reactor
global

33 34

Collaboration in the Reactor

Pattern
Structure of the Reactor Pattern
callback :
main Concrete reactor
select (handles); program Event_Handler : Reactor
Concrete
INITIALIZATION

foreach h in handles {
if (h is output handler) Event_Handler Reactor()
table[h]->handle_output () ; INITIALIZE
if (h is input handler)
table[h]->handle_input (); register_handler(callback)
MODE

AP
if (h is signal handler) PL REGISTER HANDLER
table[h]->handle_signal (); Event_Handler DE ICA
PE T
} ND ION-
EN get_handle()
this->expire_timers (); handle_input() AP
T EXTRACT HANDLE
handle_output() P
IN LICA
handle_signal()
DE
PE TION handle_events()
ND - START EVENT LOOP
handle_timeout() EN
T
n get_handle() n select()
A FOREACH EVENT DO
EVENT HANDLING

1
1 1 handle_input()
DATA ARRIVES
Reactor Timer_Queue
handle_events()
1 handle_output()
n
MODE

1 schedule_timer(h) OK TO SEND
register_handler(h) Handles cancel_timer(h)
remove_handler(h)
expire_timer(h)
handle_signal()
expire_timers() SIGNAL ARRIVES
n
1 1 handle_timeout()
TIMER EXPIRES
remove_handler(callback)
REMOVE HANDLER

CLEANUP
handle_close()

35 36
 The Acceptor Pattern
Using the Reactor Pattern in the
Logging Server  Intent
{ \Decouple the passive initialization of a service
from the tasks performed once the service is ini-
REGISTERED tialized"
APPLICATION

: Logging OBJECTS 2: sh = new Logging_Handler

LEVEL

: Logging 3: accept (*sh)

Handler 4: sh->open() : Logging
Handler
Acceptor
 This pattern resolves the following forces
for network servers using interfaces like sock-
: Event 5: handle_input()
: Event
Handler 6: recv(msg)
Handler
7:process(msg)
: Event
Handler ets or TLI:
1: handle_input()
1. How to reuse passive connection establishment code
FRAMEWORK

for each new service

LEVEL

:Timer : Handle
Table : Signal
Queue
Handlers 2. How to make the connection establishment code
: Reactor portable across platforms that may contain sock-
ets but not TLI, or vice versa
OS EVENT DEMULTIPLEXING INTERFACE
KERNEL
LEVEL

3. How to enable exible policies for creation, con-

nection establishment, and concurrency
4. How to ensure that a passive-mode descriptor is
not accidentally used to read or write data
37 38

Structure of the Acceptor Pattern Collaboration in the Acceptor

Pattern
n
APPLICATION

SOCK_Stream Concrete_Svc_Handler
1 SOCK_Acceptor
Concrete acc : peer_acceptor_ sh: reactor :
LAYER

Server : SOCK
Svc Handler Concrete Acceptor
Acceptor
Svc_Handler Reactor
open()
open()
Acceptor INITIALIZE PASSIVE open()
PROCESSING INITIALIZATION INITIALIZATION

ENDPOINT
ENDPOINT

register_handler(acc)
PHASE

REGISTER HANDLER

EXTRACT HANDLE
get_handle()
SVC_HANDLER handle_events()
A PEER_ACCEPTOR START EVENT LOOP
PEER_STREAM
Svc select()
Acceptor FOREACH EVENT DO
handle_input()
Handler
CONNECTION

CONNECTION EVENT
S sh = make_svc_handler()
INIT make_svc_handler()
SERVICE

open()
LAYER

CREATE, ACCEPT, accept_svc_handler (sh)

PHASE

accept_svc_handler() AND ACTIVATE OBJECT activate_svc_handler (sh)

A activate_svc_handler() register_handler(sh)
REGISTER HANDLER
open() FOR CLIENT I/O
get_handle()
handle_input() EXTRACT HANDLE
handle_input()
DATA EVENT
SERVICE

sh = make_svc_handler();
PHASE

accept_svc_handler (sh); PROCESS MSG

svc()
activate_svc_handler (sh);
CLIENT SHUTDOWN
handle_close()
handle_close()
REACTIVE

SERVER SHUTDOWN
Event
LAYER

Handler Reactor
handle_input() n 1

 Acceptor factory creates, connects, and ac-

tivates a Svc Handler
39 40
Using the Acceptor Pattern in the Acceptor Class Public Interface
Logging Server  A reusable template factory class that ac-
cepts connections from clients
: Logging : Logging template <class SVC_HANDLER, // Type of service
: Logging
Handler Handler
: Logging Handler class PEER_ACCEPTOR> // Accepts connections
Acceptor class Acceptor : public Service_Object {
: Svc : Svc
: Svc
Handler Handler
// Inherits from Event_Handler
Handler
: Acceptor public:
// Initialization.
ACTIVE
virtual int open (const PEER_ACCEPTOR::PEER_ADDR &);
PASSIVE : Logging
LISTENER CONNECTIONS
Handler
// Template Method or Strategy for creating,
1: handle_input() // connecting, and activating SVC_HANDLER's.
2: sh = make_svc_handler() : Svc
3: accept_svc_handler(sh) : Reactor Handler virtual int handle_input (HANDLE);
4: activate_svc_handler(sh)
// Demultiplexing hooks.
virtual HANDLE get_handle (void) const;
virtual int handle_close (HANDLE, Reactor_Mask);

41 42

Acceptor Class Protected and

Private Interfaces Acceptor Class Implementation
// Shorthand names.
#define SH SVC_HANDLER
 Only visible to the class and its subclasses #define PA PEER_ACCEPTOR

// Initialization.
protected:
// Factory method that creates a service handler.
template <class SH, class PA> int
virtual SVC_HANDLER *make_svc_handler (void);
Acceptor<SH, PA>::open (const PA::PEER_ADDR &addr)
{
// Factory method that accepts a new connection.
// Forward initialization to concrete peer acceptor
virtual int accept_svc_handler (SVC_HANDLER *);
peer_acceptor_.open (addr);
// Factory method that activates a service handler.
// Register with Reactor.
virtual int activate_svc_handler (SVC_HANDLER *);
Service_Config::reactor()->register_handler
private:
(this, Event_Handler::READ_MASK);
// Passive connection mechanism.
}
PEER_ACCEPTOR peer_acceptor_;
};

43 44

// Template Method or Strategy Factory that creates,
// connects, and activates new SVC_HANDLER objects.
template <class SH, class PA> int
Acceptor<SH, PA>::handle_input (HANDLE)
{
// Factory Method that makes a service handler.
SH *svc_handler = make_svc_handler ();
// Accept the connection.
accept_svc_handler (svc_handler);
// Delegate control to the service handler.
activate_svc_handler (svc_handler);
} Acceptor<SH, PA>::activate_svc_handler (SH *svc_handler)
45
// Returns underlying I/O descriptor (called by Reactor).
template <class SH, class PA> HANDLE
Acceptor<SH, PA>::get_handle (void) const
{
return peer_acceptor_.get_handle ();
}
// Perform termination activities when removed from Reactor.
template <class SH, class PA> int
Acceptor<SH, PA>::handle_close (HANDLE, Reactor_Mask)
{
peer_acceptor_.close ();
}
47
// Factory method for creating a service handler.
// Can be overridden by subclasses to define new
// allocation policies (such as Singletons, etc.).
template <class SH, class PA> SH *
Acceptor<SH, PA>::make_svc_handler (HANDLE)
{
return new SH; // Default behavior.
}
// Accept connections from clients (can be overridden).
template <class SH, class PA> int
Acceptor<SH, PA>::accept_svc_handler (SH *svc_handler)
{
peer_acceptor_.accept (*svc_handler);
}
// Activate the service handler (can be overridden).
template <class SH, class PA> int
{
if (svc_handler->open () == -1)
svc_handler->close ();
}
46
Svc Handler Class Public Interface
 Provides a generic interface for communi-
cation services that exchange data with a
peer over a network connection
template <class PEER_STREAM, // Communication mechanism.
class SYNCH> // Synchronization policy.
class Svc_Handler : public Task<SYNCH>
{
public:
Svc_Handler (void); // Constructor.
// Activate the client handler.
virtual int open (void *);
// Returns underlying PEER_STREAM.
operator PEER_STREAM &();
// Overloaded new operator that detects
// when a Svc_Handler is allocated dynamically.
void *operator new (size_t n);
// Return underlying IPC mechanism.
PEER_STREAM &peer (void);
48
 Svc Handler Class Protected
Interface
 By default, a Svc Handler object is registered
 Contains the demultiplexing hooks and other
implementation artifacts
protected:
// Demultiplexing hooks inherited from Task.
virtual int handle_close (HANDLE, Reactor_Mask);
virtual HANDLE get_handle (void) const;
virtual void set_handle (HANDLE);
private:
PEER_STREAM peer_; // IPC mechanism.
Svc Handler implementation
with the Reactor
{ This makes the service singled-threaded and no
other synchronization mechanisms are necessary
#define PS PEER_STREAM // Convenient short-hand.
template <class PS, class SYNCH> int
Svc_Handler<PS, SYNCH>::open (void *)
{
// Enable non-blocking I/O.
peer ().enable (ACE_NONBLOCK);

// Records if we're allocated dynamically. // Register handler with the Reactor.

int is_dynamic_; Service_Config::reactor()->register_handler
(this, Event_Handler::READ_MASK);
virtual ~Svc_Handler (void); }
};

49 50

// Extract the underlying I/O descriptor. // Perform termination activities and

// deletes memory if we're allocated dynamically.
template <class PS, class SYNCH> HANDLE
Svc_Handler<PS, SYNCH>::get_handle (void) const template <class PS, class SYNCH> int
{ Svc_Handler<PS, SYNCH>::handle_close
// Forward request to underlying IPC mechanism. (HANDLE, Reactor_Mask)
return peer ().get_handle (); {
} peer ().close ();

// Set the underlying I/O descriptor. if (is_dynamic_)

// This must be allocated dynamically!
template <class PS, class SYNCH> void delete this;
Svc_Handler<PS, SYNCH>::set_handle (HANDLE h)
{ return 0;
peer ().set_handle (h); }
}
// Extract the underlying PEER_STREAM (used by
// Return underlying IPC mechanism. // Acceptor::accept() and Connector::connect()).

template <class PS, class SYNCH> PS & template <class PS, class SYNCH>
Svc_Handler<PS, SYNCH>::peer (void) Svc_Handler<PS, SYNCH>::operator PS &()
{ {
return peer_; return peer_;
} }

51 52
 OO Design Interlude Object Diagram for OO Logging
Server
 Q: \How can we determine if an object is
allocated dynamically?" SERVER
LOGGING : Service
DAEMON
Config

 A: Overload class-specic operator new and : Logging

Acceptor : Service
use thread-specic storage to associate the : Logging
Handler
Repository
: Service
pointers, e.g., : Logging
Manager

Handler : Reactor
template <class PS, class SYNCH> void *
Svc_Handler<PS, SYNCH>::operator new (size_t n)
{
void *allocated = ::new char[n]; LOGGING
CONNECTION SERVER REMOTE
RECORDS
set_thread_specific (key_, allocated); REQUEST CONTROL
return allocated; OPERATIONS
}

template <class PS, class SYNCH> CLIENT

Svc_Handler<PS, SYNCH>::Svc_Handler (void) CLIENT CLIENT
{ CLIENT
void *allocated = get_thread_specific (key_);
is_dynamic_ = allocated == this;
}

53 54

The Logging Handler and OO Design Interlude

Logging Acceptor Classes
A

 These instantiated template classes imple- IPC_SAP

ment application-specic server logging dae-

mon functionality SOCK_SAP TLI_SAP SPIPE_SAP FIFO_SAP

// Performs I/O with client logging daemons. TRANSPORT

SOCKET STREAM PIPE NAMED PIPE
LAYER
API API API
INTERFACE API
class Logging_Handler :
public Svc_Handler<SOCK_Stream, NULL_SYNCH> {
public:
// Recv and process remote logging records.
virtual int handle_input (HANDLE);
protected:  Q: What are the SOCK * classes and why
};
char host_name_[MAXHOSTNAMELEN + 1]; // Client.
are they used rather than using sockets di-
rectly?
// Logging_Handler factory.

class Logging_Acceptor :  A: SOCK * are \wrappers" that encapsulate

public Acceptor<Logging_Handler, SOCK_Acceptor> { network programming interfaces like sock-
public:
// Dynamic linking hooks.
ets and TLI
virtual int init (int argc, char *argv[]);
virtual int fini (void);
{ This is an example of the \Wrapper pattern"
};
55 56
 Structure of the Wrapper Pattern
The Wrapper Pattern
1: request ()
 Intent
client Wrapper
{ \Encapsulate lower-level functions within type-safe,
modular, and portable class interfaces" request()
 This pattern resolves the following forces 2: specific_request()
that arise when using native C-level OS APIs
1. How to avoid tedious, error-prone, and non-portable
programming of low-level IPC mechanisms Wrappee
specific_request()
2. How to combine multiple related, but independent,
functions into a single cohesive abstraction
57 58
Socket Structure
Socket Taxonomy
gethostbyname()
getservbyname()
getsockname()

COMMUNICATION DOMAIN
getpeername()

ON
getsockopt()

LOCAL/REMOTE
setsockopt()

TI E LOCAL
recvfrom()

EC IV
sendmsg()
recvmsg()

NN LE PASS
connect()

CO RO
sendto()
socket()

accept()

writev()
readv()
listen()

write()

send()
bind()

read()

recv()

E
TIV
AC

TYPE OF COMMUNICATION SERVICE

socket(PF_UNX) socket(PF_INET)
bind() recvfrom() bind() recvfrom()

GRAM
DATA
socket(PF_UNIX) socket(PF_INET)
bind() sendto() bind() sendto()
Note that this API is linear rather than hi-

CONNECTED
DATAGRAM
socket(PF_UNIX) socket(PF_INET)

erarchical bind() connect() recv()
socket(PF_UNIX)
bind() connect() recv()
socket(PF_INET)
bind() connect() send() bind() connect() send()
{ Thus, it gives no hints on how to use it correctly accept(PF_UNIX) accept(PF_INET)

STREAM
listen() send()/recv() listen() send()/recv()
socket(PF_UNIX) bind() socket(PF_INET) bind()
connect() send()/recv() connect() send()/recv()
 In addition, there is no consistency among
names :::
59 60
 SOCK SAP Factory Class
SOCK SAP Class Structure Interfaces
class SOCK_Connector : public SOCK
{
COMMUNICATION DOMAIN
N public:
IO LOCAL LOCAL/REMOTE
CT E // Traits
NNE E SS
IV
L typedef INET_Addr PEER_ADDR;
CO RO PA
E typedef SOCK_Stream PEER_STREAM;
TIV
AC
int connect (SOCK_Stream &new_sap, const Addr &remote_addr,
TYPE OF COMMUNICATION SERVICE

LSOCK_Dgram SOCK_Dgram Time_Value *timeout, const Addr &local_addr);

GRAM
DATA

SOCK_Dgram // ...
LSOCK_Dgram SOCK_Dgram_Bcast };
SOCK_Dgram_Mcast
CONNECTED
DATAGRAM

LSOCK_Dgram SOCK_Dgram
class SOCK_Acceptor : public SOCK
{
LSOCK_CODgram SOCK_CODgram public:
// Traits
LSOCK_Acceptor SOCK_Acceptor
typedef INET_Addr PEER_ADDR;
STREAM

LSOCK_Stream SOCK_Stream
typedef SOCK_Stream PEER_STREAM;
LOCK_Connector SOCK_Connector
LSOCK_Stream SOCK_Stream
SOCK_Acceptor (const Addr &local_addr);

int accept (SOCK_Stream &, Addr *, Time_Value *) const;

//...
};

61 62

SOCK SAP Stream and OO Design Interlude

Addressing Class Interfaces
class SOCK_Stream : public SOCK
{
 Q: Why decouple the SOCK Acceptor and
public: the SOCK Connector from SOCK Stream?
typedef INET_Addr PEER_ADDR; // Trait.

ssize_t send (const void *buf, int n);

 A: For the same reasons that Acceptor and
ssize_t recv (void *buf, int n);
Connector are decoupled from Svc Handler,
ssize_t send_n (const void *buf, int n);
ssize_t recv_n (void *buf, int n);
e.g.,
ssize_t send_n (const void *buf, int n, Time_Value *);
{ A SOCK Stream is only responsible for data trans-
ssize_t recv_n (void *buf, int n, Time_Value *);
int close (void);
fer
// ...
}; . Regardless of whether the connection is estab-
lished passively or actively
class INET_Addr : public Addr
{ { This ensures that the SOCK* components are never
public:
INET_Addr (u_short port_number, const char host[]);
used incorrectly: : :
u_short get_port_number (void);
. e.g., you can't accidentally read or write on
SOCK Connectors or SOCK Acceptors, etc.
int32 get_ip_addr (void);
// ...
};

63 64
 SOCK SAP Hierarchy OO Design Interlude
A A

IPC
SAP
SOCK
 Q: \How can you switch between dierent
IPC mechanisms?"
SOCK SOCK SOCK SOCK SOCK SOCK
Dgram Dgram CODgram Stream Connector Acceptor
Bcast

 A: By parameterizing IPC Mechanisms with

SOCK
Dgram
Mcast
LSOCK
Dgram
LSOCK
CODgram
LSOCK
Stream
LSOCK
Connector
LSOCK
Acceptor C++ Templates!
GROUP DATAGRAM A STREAM CONNECTION
COMM COMM
LSOCK
COMM ESTABLISHMENT
#if defined (ACE_USE_SOCKETS)
typedef SOCK_Acceptor PEER_ACCEPTOR;
#elif defined (ACE_USE_TLI)
typedef TLI_Acceptor PEER_ACCEPTOR;
#endif /* ACE_USE_SOCKETS */

class Logging_Handler : public

 Shared behavior is isolated in base classes Svc_Handler<PEER_ACCEPTOR::PEER_STREAM,
NULL_SYNCH>
{ /* ... /* };

 Derived classes implement dierent com- class Logging_Acceptor : public

munication services, communication domains, Acceptor <Logging_Handler, PEER_ACCEPTOR>
{ /* ... */ };
and connection roles
65 66

// Automatically called when a Logging_Acceptor object

// is dynamically linked.

Logging_Acceptor::init (int argc, char *argv[])

Logging Handler Implementation {

Get_Opt get_opt (argc, argv, "p:", 0);
INET_Addr addr;

 Implementation of the application-specic for (int c; (c = get_opt ()) != -1; )

switch (c)
logging method {
case 'p':
addr.set (atoi (getopt.optarg));
// Callback routine that receives logging records. break;
// This is the main code supplied by a developer! default:
break;
template <class PS, class SYNCH> int }
Logging_Handler<PS, SYNCH>::handle_input (HANDLE)
{ // Initialize endpoint and register with the Reactor
// Call existing function to recv open (addr);
// logging record and print to stdout. }
handle_log_record (peer ().get_handle (), 1);
} // Automatically called when object is dynamically unlinked.

Logging_Acceptor::fini (void)
{
handle_close ();
}

67 68
 Structure of the Service
The Service Congurator Pattern Congurator Pattern
 Intent

APPLICATION
Concrete

LAYER
{ \Decouple the behavior of network services from Service Object
the point in time at which these services are con-
gured into an application"

 This pattern resolves the following forces Service Service

for highly exible communication software: Object Config

CONFIGURATION
suspend() 1

{ How to defer the selection of a particular type, or resume() 1

LAYER
a particular implementation, of a service until very init() A
fini()
late in the design cycle info() n
1

Service
1
. i.e., at installation-time or run-time Repository

{ How to build complete applications by composing

multiple independently developed services

REACTIVE
LAYER
Event
Handler n Reactor
{ How to recongure and control the behavior of the 1
service at run-time

69 70

Using the Service Congurator

Collaboration in the Service Pattern for the Logging Server
Congurator Pattern
svc : : Service : Service SERVICE : Thread
main()
Service_Object
: Reactor
Config Repository CONFIGURATOR
Logger
Service_Config() RUNTIME
: Reactive
CONFIGURE
FOREACH SVC ENTRY DO process_directives() Logger : Service
Object
CONFIGURATION

DYNAMICALLY LINK link_service() SHARED

SERVICE
init(argc, argv) : Service : Service OBJECTS
MODE

INITIALIZE SERVICE
register_handler(svc) Repository Object
REGISTER SERVICE
get_handle()
EXTRACT HANDLE : Thread Pool
STORE IN REPOSITORY
insert() Logger
START EVENT LOOP
run_event_loop() : Service : Reactor
handle_events() Config : Service
EVENT HANDLING

FOREACH EVENT DO

INCOMING EVENT
handle_input() Object
MODE

SHUTDOWN EVENT handle_close()

CLOSE SERVICE remove_handler(svc)

UNLINK SERVICE
unlink_service()
fini() remove()

 Existing service is single-threaded, other ver-

sions could be multi-threaded :::

71 72
 Service Conguration
Dynamic Linking a Service
 The logging service is congured and ini-
 Application-specic factory function used to tialized based upon the contents of a con-
dynamically create a service guration script called svc.conf:
% cat ./svc.conf
// Dynamically linked factory function that allocates # Dynamically configure the logging service
// a new Logging_Acceptor object dynamically dynamic Logger Service_Object *
logger.dll:make_Logger() "-p 2010"
extern "C" Service_Object *make_Logger (void);

Generic event-loop to dynamically congure

Service_Object *
make_Logger (void) 
{
return new Logging_Acceptor;
service daemons
// Framework automatically deletes memory.
} int
main (int argc, char *argv[])
{

 The make Logger function provides a hook Service_Config daemon;

between an application-specic service and // Initialize the daemon and configure services

the application-independent ACE mechanisms daemon.open (argc, argv);

// Run forever, performing configured services

{ ACE handles all memory allocation and dealloca- daemon.run_reactor_event_loop ();
tion /* NOTREACHED */
}

73 74

State-chart Diagram for the Collaboration of Patterns in the

Service Congurator Pattern Server Logging Daemon
CONFIGURE/ Logger
A: C:
: Service : Service
Service_Config::
Service_Config::process_directives
process_directives()
() Daemon
Logging Logging
Config
: Reactor
Repository
Acceptor Handler
Service_Config()
CONFIGURE
process_directives()
INITIALIZED FOREACH SVC ENTRY DO
link_service()
LINK SERVICE
init(argc, argv)
IDLE INITIALIZE SERVICE
register_handler(A)
START EVENT LOOP/ REGISTER SERVICE
get_handle()
Service_Config::
Service_Config::run_event_loop
run_event_loop()
() EXTRACT HANDLE
insert()
STORE IN REPOSITORY

run_event_loop()
SHUTDOWN/ NETWORK EVENT/
START EVENT LOOP
handle_events()
Service_Config::
Service_Config::close
close()
() AWAITING FOREACH EVENT DO
EVENTS Reactor::
Reactor::dispatch
dispatch()
() CONNECTION EVENT
handle_input()
C = new Logging_Handler
ALLOCATE AND accept (C);
ACTIVATE OBJECT
C->open(A)
REGISTER HANDLER register_handler(C)
PERFORM FOR CLIENT I/O get_handle()
RECONFIGURE/ CALLBACK EXTRACT HANDLE
handle_input()
Service_Config::
Service_Config::process_directives
process_directives()
() DATA EVENT
write()
PROCESS LOGGING

CALL HANDLER/ RECORD

handle_close()
Event_Handler::
Event_Handler::handle_input
handle_input()
() CLIENT SHUTDOWN remove_handler(C)
handle_close()
DAEMON SHUTDOWN remove_handler(A)

 Note the separation of concerns between UNLINK SERVICE fini()

unlink_service()

objects
remove()
:::

75 76
 Advantages of OO Logging
Server Concurrent OO Logging Server
 The OO architecture illustrated thus far de-
couples application-specic service function- The structure of the server logging daemon
ality from: 
can benet from concurrent execution on a
* Time when a service is congured into a process
* The number of services per-process
* The type of IPC mechanism used
* The type of event demultiplexing mechanism used
 We can use the techniques discussed thus
far to extend applications without:
1. Modifying, recompiling, and relinking existing code
multi-processor platform
 This section examines ACE C++ classes
and design patterns that extend the logging
server to incorporate concurrency
{ Note how most extensions require minimal changes
to the existing OO architecture: : :

2. Terminating and restarting executing daemons

 This example also illustrates additional ACE
components involving synchronization and
 The remainder of the slides examine a set of multi-threading
techniques for decoupling functionality from
concurrency mechanisms, as well
77 78

Concurrent OO Logging Server Pseudo-code for Concurrent

Architecture Server
SERVER SERVER  Pseudo-code for multi-threaded Logging Handler
LOGGING DAEMON factory server logging daemon
: logging
: logging
handler
: logging
handler void handler factory (void)
acceptor
f
initialize listener endpoint
CONNECTION foreach (pending connection request) f
CLIENT REQUEST accept request
spawn a thread to handle request
call logger handler() active object
g
LOGGING
LOGGING g
RECORDS
RECORDS
 Pseudo-code for server logging daemon ac-
CLIENT
NETWORK CLIENT
tive object
void logging handler (void)
f
foreach (incoming logging records from client)
call handle log record()
 Thread-based implementation exit thread
g
79 80
Application-specic Logging Code Class Diagram for Concurrent
OO Logging Server
 The OO implementation localizes the application- Thr_Logging_Handler SOCK_Stream
specic part of the logging service in a sin-

APPLICATION-

COMPONENTS
SOCK_Acceptor n NULL_SYNCH
1

SPECIFIC
gle point, while leveraging o reusable ACE Thr
Logging
Thr
Logging
components Acceptor INITS
Handler

// Handle all logging records from a particular client

CONNECTION-
SVC_HANDLER

COMPONENTS
// (run in each slave thread). PEER_STREAM

ORIENTED
PEER_ACCEPTOR
SYNCH
int Acceptor Svc
Thr_Logging_Handler::svc (void) Handler
{
// Perform a "blocking" receive and process logging
// records until the client closes down the connection.
PEER PEER
ACCEPTOR STREAM
// Danger! race conditions...

COMPONENTS
FRAMEWORK
Stream
for (; ; ++request_count) Connection

ACE
handle_log_record (get_handle (), 1);
IPC_SAP
Service
/* NOTREACHED */ Configurator
return 0;
} Concurrency Reactor
global

81 82

Task Inheritance Hierarchy

ACE Tasks SYNCH SYNCH
Service Task
 An ACE Task binds a separate thread of open()=0
control together with an object's data and close()=0
methods
N
-

O T
TI EN put()=0
A
N IC ND svc()=0
-

O
TI C PL PE A
{ Multiple active objects may execute in parallel in ICA IFI
PL EC
AP DE
IN
separate lightweight or heavyweight processes AP SP
SYNCH
Service Message
Event Object Queue
 Task objects communicate by passing typed Handler suspend()=0
messages to other Tasks handle_input()
resume()=0
A
{ Each Task maintains a queue of pending messages handle_output() Shared
handle_exception()
that it processes in priority order handle_signal()
Object
handle_timeout () init()=0
handle_close() fini ()=0
get_handle()=0 info()=0
 ACE Task are a low-level mechanism to sup- A A

port \active objects"

 Supports dynamically congured services
83 84
 Task Class Public Interface
 C++ interface for message processing
* Tasks can register with a Reactor
* They can be dynamically linked
* They can queue data
* They can run as \active objects"
 e.g.,
template <class SYNCH>
class Task : public Service_Object
{ Thread_Manager *thr_mgr (void);
public:
// Initialization/termination routines.
virtual int open (void *args = 0) = 0;
virtual int close (u_long flags = 0) = 0;
// Transfer msg to queue for immediate processing.
virtual int put (Message_Block *, Time_Value * = 0) = 0;
// Run by a daemon thread for deferred processing.
virtual int svc (void) = 0;
// Turn the task into an active object.
int activate (long flags);
85
OO Design Interlude
 Q: What is the svc run() function and why
is it a static method?
 A: OS thread spawn APIs require a C-style
function as the entry point into a thread
 The Stream class category encapsulates the
svc run function within the Task::activate
method:
template <class SYNCH> int
Task<SYNCH>::activate (long flags, int n_threads)
{
if (thr_mgr () == NULL)
thr_mgr (Service_Config::thr_mgr ());
thr_mgr ()->spawn_n
(n_threads, &Task<SYNCH>::svc_run,
(void *) this, flags);
}
87
Task Class Protected Interface
 The following methods are mostly used within
put and svc
// Accessors to internal queue.
Message_Queue<SYNCH> *msg_queue (void);
void msg_queue (Message_Queue<SYNCH> *);
// Accessors to thread manager.
void thr_mgr (Thread_Manager *);
// Insert message into the message list.
int putq (Message_Block *, Time_Value *tv = 0);
// Extract the first message from the list (blocking).
int getq (Message_Block *&mb, Time_Value *tv = 0);
// Hook into the underlying thread library.
static void *svc_run (Task<SYNCH> *);
86
OO Design Interlude (cont'd)
 Task::svc run is static method used as the
entry point to execute an instance of a ser-
vice concurrently in its own thread
template <class SYNCH> void *
Task<SYNCH>::svc_run (Task<SYNCH> *t)
{
Thread_Control tc (t->thr_mgr ()); // Record thread ID.
// Run service handler and record return value.
void *status = (void *) t->svc ();
tc.status (status);
t->close (u_long (status));
// Status becomes `return' value of thread...
return status;
// Thread removed from thr_mgr()
// automatically on return...
}
88
 OO Design Interlude
The Active Object Pattern
 Q: \How can groups of collaborating threads  Intent
be managed atomically?"
{ \Decouple method execution from method invo-
cation and simplies synchronized access to shared
 A: Develop a \thread manager" class resources by concurrent threads"

{ Thread Manager is a collection class  This pattern resolves the following forces
. It provides mechanisms for suspending and re- for concurrent communication software:
suming groups of threads atomically { How to allow blocking read and write operations
on one endpoint that do not detract from the qual-
. It implements barrier synchronization on thread ity of service of other endpoints
exits
{ How to serialize concurrent access to shared object
{ Thread Manager also shields applications from in- state
compabitilities between dierent OS thread libraries
{ How to simplify composition of independent ser-
. It is integrated into ACE via the Task::activate vices
method

89 90

Structure of the Active Object

Pattern
Collaboration in the Active
Client
loop {
m = actQueue.remove()
dispatch (m)
Object Pattern
Interface
}
ResultHandle m1()
ResultHandle m2() : Client : Activation : Represent-
Scheduler client
Interface
: Scheduler
Queue ation
ResultHandle m3()
METHOD OBJECT
CONSTRUCTION

dispatch() m1()
m1'() Activation INVOKE

cons(m1')
m2'() 1 Queue CREATE METHOD
OBJECT
VISIBLE m3'() 1 future()
insert() RETURN RESULT
TO HANDLE
remove()
CLIENTS 1
SCHEDULING/

1
EXECUTION

INSERT IN insert(m1')
PRIORITY QUEUE
1 n
INVISIBLE DEQUEUE NEXT remove(m1')
METHOD OBJECT
TO Resource Method
CLIENTS Representation Objects
COMPLETION

EXECUTE dispatch(m1')
RETURN RESULT reply_to_future()

 The Scheduler is a \meta-object" that de-

termines the sequence that Method Objects
are executed
91 92
 ACE Task Support for Active
Objects Collaboration in ACE Active
Objects
t2 : Task t1 : t2 : t2msg_q : t3 :

PRODUCER
Task Task Message_Queue Task

PHASE
2: enqueue (msg) 3: svc () work()
4: dequeue (msg) PRODUCER TASK
: Message 5: do_work(msg)
Queue
PASS MSG put(msg)
ACTIVE
6: put (msg) enqueue_tail(msg)

QUEUEING
ENQUEUE MSG
1: put (msg)

PHASE
RUN SVC()
svc()
ASYNCHRONOUSLY
t3 : Task dequeue_head(msg)
DEQUEUE MSG
t1 : Task

CONSUMER
: Message

PHASE
Queue CONSUMER TASK work()
: Message
Queue ACTIVE
PASS MSG put(msg)
ACTIVE

93 94

Thr Logging Acceptor and

Thr Logging Handler
 Template classes that create, connect, and // Override definition in the Svc_Handler class
activate a new thread to handle each client // (spawns a new thread in this case!).

int
class Thr_Logging_Handler Thr_Logging_Handler::open (void *)
: public Svc_Handler<SOCK_Stream, NULL_SYNCH> {
{ INET_Addr client_addr;
public:
// Override definition in the Svc_Handler class peer ().get_local_addr (client_addr);
// (spawns a new thread!). strncpy (host_name_, client_addr.get_host_name (),
virtual int open (void *); MAXHOSTNAMELEN + 1);

// Process remote logging records. // Spawn a new thread to handle

virtual int svc (void); // logging records with the client.
}; activate (THR_BOUND | THR_DETACHED);
}
class Thr_Logging_Acceptor :
public Acceptor<Thr_Logging_Handler,
SOCK_Acceptor>
{
// Same as Logging_Acceptor...
};

95 96
// Process remote logging records.
Dynamically Reconguring the
int Logging Server
Thr_Logging_Handler::svc (void)
{
// Loop until the client terminates the connection.
 The concurrent logging service is congured
for (; ; ++request_count) and initialized by making a small change to
the svc.conf le:
// Call existing function to recv logging
// record and print to stdout.

handle_log_record (peer ().get_handle (), 1);

% cat ./svc.conf
# Dynamically configure the logging service
/* NOTREACHED */
# dynamic Logger Service_Object *
return 0;
# /svcs/logger.dll:make_Logger() "-p 2010"
}
remove Logger
dynamic Logger Service_Object *
// Dynamically linked factory function that
thr_logger.dll:make_Logger() "-p 2010"
// allocates a new threaded logging Acceptor object.

extern "C" Service_Object *make_Logger (void);

 Sending a SIGHUP signal recongures the
Service_Object * server logger daemon process
make_Logger (void)
{ { The original sequential version of the Logger ser-
return new Thr_Logging_Acceptor; vice is dynamically unlinked and replaced with the
} new concurrent version

97 98

Caveats Explicit Synchronization

Mechanisms
 The concurrent server logging daemon has
several problems
1. Output in the handle log record function is not
 One approach for serialization uses OS mu-
serialized tual exclusion mechanisms explicitly, e.g.,
2. The auto-increment of global variable request count // at file scope
is also not serialized mutex_t lock; // SunOS 5.x synchronization mechanism

// ...
handle_log_record (HANDLE in_h, HANDLE out_h)
 Lack of serialization leads to errors on many {
shared memory multi-processor platforms :::
// in method scope ...
mutex_lock (&lock);

{ Note that this problem is indicative of a large class write (out_h, log_record.buf, log_record.size);

of errors in concurrent programs: : : mutex_unlock (&lock);

// ...
}

 The following slides compare and contrast a  However, adding these mutex calls explicitly
series of techniques that address this prob- is inelegant, tedious, error-prone, and non-
lem portable
99 100
 C++ Wrappers for
Synchronization Porting Mutex to Windows NT
 To address portability problems, dene a  WIN32 version of Mutex
C++ wrapper: class Mutex
{
class Mutex
public:
{
Mutex (void) {
public:
lock_ = CreateMutex (0, FALSE, 0);
Mutex (void) {
}
mutex_init (&lock_, USYNCH_THREAD, 0);
~Mutex (void) {
}
CloseHandle (lock_);
~Mutex (void) { mutex_destroy (&lock_); }
}
int acquire (void) { return mutex_lock (&lock_); }
int acquire (void) {
int tryacquire (void) { return mutex_trylock (&lock); }
return WaitForSingleObject (lock_, INFINITE);
int release (void) { return mutex_unlock (&lock_); }
}
int tryacquire (void) {
private:
return WaitForSingleObject (lock_, 0);
mutex_t lock_; // SunOS 5.x serialization mechanism.
}
void operator= (const Mutex &);
int release (void) {
void Mutex (const Mutex &);
return ReleaseMutex (lock_);
};
}
private:
HANDLE lock_; // Win32 locking mechanism.
 Note, this mutual exclusion class interface // ...

is portable to other OS platforms

101 102

Using the C++ Mutex Wrapper Automated Mutex Acquisition and

Release
 Using C++ wrappers improves portability
and elegance  To ensure mutexes are locked and unlocked,
we'll dene a template class that acquires
// at file scope
and releases a mutex automatically
Mutex lock; // Implicitly "unlocked".
template <class LOCK>
// ... class Guard
handle_log_record (HANDLE in_h, HANDLE out_h) {
{ public:
// in method scope ... Guard (LOCK &m): lock (m) { lock_.acquire (); }
~Guard (void) { lock_.release (); }
lock.acquire ();
write (out_h, log_record.buf, log_record.size); private:
lock.release (); LOCK &lock_;
}
// ...
}

 Guard uses the C++ idiom whereby a con-

 However, this doesn't really solve the te- structor acquires a resource and the destruc-
dium or error-proneness problems tor releases the resource
103 104
 OO Design Interlude
 Q: Why is Guard parameterized by the type The Adapter Pattern
of LOCK?
 Intent
 A: since there are many dierent
avors of { \Convert the interface of a class into another in-
locking that benet from the Guard func-
tionality, e.g.,
* Non-recursive vs recursive mutexes
* Intra-process vs inter-process mutexes
* Readers/writer mutexes
* Solaris and System V semaphores
* File locks
* Null mutex
 In ACE, all synchronization wrappers use to
Adapter pattern to provide identical inter-
faces whenever possible to facilitate param-
eterization
terface client expects"
. Adapter lets classes work together that couldn't
otherwise because of incompatible interfaces
 This pattern resolves the following force that
arises when using conventional OS inter-
faces
1. How to provide an interface that expresses the
similarities of seemingly dierent OS mechanisms
(such as locking or IPC)

105 106

Using the Adapter Pattern for

Locking
Structure of the Adapter Pattern
LOCK
1: request () client
Guard
Adapter Guard()
client 1: Guard() ~Guard()
request()
Mutex
2: specific_request()
Adaptee1 Guard Mutex
Guard()
specific_request()Adaptee2 ~Guard() acquire()
2: acquire()
specific_request() Adaptee3
3: mutex_lock()
specific_request()
POSIX Win32
pthread_mutex Solaris EnterCritical
_lock() Section()
mutex_lock()

107 108
A thread-safe handle log record() Remaining Caveats
Function
template <class LOCK = Mutex> ssize_t
handle_log_record (HANDLE in_h, HANDLE out_h)  There is a race condition when incrementing
{
// new code (beware of static initialization...)
the request count variable
static LOCK lock;
ssize_t n; // Danger! race conditions...
size_t len; for (; ; ++request_count)
Log_Record log_record; handle_log_record (get_handle (), 1);

n = recv (h, (char *) &len, sizeof len, 0);

if (n != sizeof len) return -1;
len = ntohl (len); // Convert byte-ordering
for (size_t nread = 0; nread < len; nread += n
n = recv (in_h, ((char *) &log_record) + nread,
len - nread, 0));
 Solving this problem using the Mutex or Guard
classes is still tedious, low-level, and error-
prone

A more elegant solution incorporates pa-

// Perform presentation layer conversions.
decode (&log_record); 
// Automatically acquire mutex lock.
Guard<LOCK> monitor (lock);
rameterized types, overloading, and the Dec-
write (out_h, log_record.buf, log_record.size); orator pattern
// Automatically release mutex lock.
}

109 110

Transparently Parameterizing Final Version of Concurrent

Synchronization Using C++ Logging Server
 The following C++ template class uses the  Using the Atomic Op class, only one change
\Decorator" pattern to dene a set of atomic is made
operations on a type parameter:
// At file scope.
template <class LOCK = Mutex, class TYPE = u_long> typedef Atomic_Op<> COUNTER; // Note default parameters...
class Atomic_Op { COUNTER request_count;
public:
Atomic_Op (TYPE c = 0) { count_ = c; }
TYPE operator++ (void) {  request count is now serialized automatically
Guard<LOCK> m (lock_); return ++count_;
}
void operator= (const Atomic_Op &ao) { for (; ; ++request_count) // Atomic_Op::operator++
if (this != &ao) { handle_log_record (get_handle (), 1);
Guard<LOCK> m (lock_); count_ = ao.count_;
}
}  The original non-threaded version may be
operator TYPE () {
Guard<LOCK> m (lock_);
supported eciently as follows:
return count_;
} typedef Atomic_Op<Null_Mutex> COUNTER;
// Other arithmetic operations omitted... //...
private: for (; ; ++request_count)
LOCK lock_; handle_log_record<Null_Mutex>
TYPE count_; (get_handle (), 1);
};
111 112
 Synchronization-aware Logging
Classes Thread-safe handle log record
Method
 A more sophisticated approach would add template <class PS, class LOCK, class COUNTER> ssize_t

several new parameters to the Logging Handler Logging_Handler<PS, LOCK, COUNTER>::handle_log_record

(HANDLE out_h)
class {
ssize_t n;
size_t len;
template <class PEER_STREAM, Log_Record log_record;
class SYNCH, class COUNTER>
class Logging_Handler ++request_count_; // Calls COUNTER::operator++().
: public Svc_Handler<PEER_STREAM, SYNCH>
{ n = peer ().recv (&len, sizeof len);
public:
Logging_Handler (void); if (n != sizeof len) return -1;
// Process remote logging records. len = ntohl (len); // Convert byte-ordering
virtual int svc (void);
peer ().recv_n (&log_record, len);
protected:
// Receive the logging record from a client. // Perform presentation layer conversions
ssize_t handle_log_record (HANDLE out_h); log_record.decode ();
// Lock used to serialize access to std output. // Automatically acquire mutex lock.
static SYNCH::MUTEX lock_; Guard<LOCK> monitor (lock_);
// Count the number of logging records that arrive. write (out_h, log_record.buf, log_record.size);
static COUNTER request_count_; // Automatically release mutex lock.
// Name of the host we are connected to. }
char host_name_[MAXHOSTNAMELEN + 1];
};
113 114

Using the Thread-safe Application-level Gateway

handle log record() Method Example
 In order to use the thread-safe version, all  The distributed logger is a relatively simple
we need to do is instantiate with Atomic Op application
{ e.g., it doesn't to deal with network output or
typedef Logging_Handler<TLI_Stream,
NULL_SYNCH,
queueing
Atomic_Op<> >
LOGGING_HANDLER;
 The next example explores the design pat-
terns and reusable framework components
To obtain single-threaded behavior requires used in an OO architecture for application-
 level Gateways
a simple change:
{ These Gateways route messages between Peers in
typedef Logging_Handler<TLI_Stream,
a large-scale telecommunication system
NULL_SYNCH,
Atomic_Op <Null_Mutex, u_long> > { Peers and Gateways are connected via TCP/IP
LOGGING_HANDLER;

115 116
 Physical Architecture of the
Gateway OO Software Architecture of the
TRACKING
Gateway
SATELLITES STATION
PEERS
: Routing
: Output Table : Input
Channel Channel

: Reactor

STATUS INFO
: Input : Output
Channel : Channel : Channel Channel
WIDE AREA BULK DATA Acceptor
Connector
NETWORK TRANSFER
COMMANDS

INCOMING GATEWAY
GATEWAY MESSAGES
OUTGOING
MESSAGES
CONNECTION
CONNECTION
REQUEST
REQUEST

LOCAL AREA NETWORK

GROUND
STATION
PEERS

117 118

Gateway Behavior
Design Patterns in the Gateway
 Components in the Gateway behave as fol-
lows:
Active Object Router
1. Gateway parses conguration les that specify which
Peers to connect with and which routes to use
Connector Acceptor
2. Channel Connector connects to Peers, then cre-
ates and activates the appropriate Channel sub- Half-Sync/
classes (Input Channel or Output Channel) Half-Async
Service
Configurator
3. Once connected, Peers send messages to the Gate-
way STRATEGIC
PATTERNS
Reactor
{ Messages are handled by the appropriate Input Channel
TACTICAL
Template Factory
{ Input Channels work as follows: PATTERNS Iterator Method Method Proxy

(a) Receive and validate messages

(b) Consult a Routing Table  The Gateway components are based upon
(c) Forward messages to the appropriate Peer(s) a system of design patterns
via Output Channels
119 120
 Design Patterns in the Gateway
(cont'd) Using the Reactor Pattern for the
 The Gateway uses the same system of pat- Single-Threaded Gateway
terns as the distributed logger
REGISTERED
{ i.e., Reactor, Service Congurator, Active Object,

APPLICATION
and Acceptor : Output : Output
OBJECTS

LEVEL
Channel Channel 4: send(msg) : Input
Channel
2: recv(msg)
 It also contains following patterns: : Event
Handler
: Event
Handler
3: route(msg)
: Event
Handler
{ Connector pattern
Decouple the active initialization of a service 1: handle_input()

FRAMEWORK
.
from the tasks performed once the service is ini-

LEVEL
tialized :Timer : Handle
: Signal
Queue Table
{ Router pattern Handlers
: Reactor
. Decouple multiple sources of input from multiple
sources of output to prevent blocking in a single- OS EVENT DEMULTIPLEXING INTERFACE

thread of control

KERNEL
LEVEL
{ Half-Sync/Half-Async
. Decouple synchronous I/O from asynchronous
I/O in a system to simplify concurrent program-
ming eort without degrading execution eciency
121 122

Class Diagram for OO Gateway Architecture

Single-Threaded Gateway  The Gateway is decomposed into compo-
nents that are layered as follows:
Channel SOCK_Stream 1. Application-specic components
APPLICATION-

COMPONENTS

SOCK_Connector Null_Synch
1 n
SPECIFIC

Channel Input/Output { route messages among Peers

Channels
Connector Channels
ACTIVATES 2. Connection-oriented application components
{ Svc Handler
CONNECTION-

COMPONENTS

SVC_HANDLER PEER_STREAM
ORIENTED

PEER_CONNECTOR
Svc
SYNCH
. Performs I/O-related tasks with connected clients
Connector Handler
{ factory
Connector

PEER
. Establishes new connections with clients
PEER
CONNECTOR
. Dynamically creates a Svc Handler object for
STREAM

each client and \activates" it

COMPONENTS
FRAMEWORK

Stream
Connection
ACE

3. Application-independent ACE framework compo-

IPC_SAP
Service
nents
Configurator
{ Perform IPC, explicit dynamic linking, event de-
Concurrency Reactor multiplexing, event handler dispatching, multi-
global
threading, etc.
123 124
 The Connector Pattern Structure of the Connector
Pattern
 Intent Concrete_Svc_Handler
n SOCK_Stream 1

APPLICATION
SOCK_Connector
{ \Decouple the active initialization of a service from Concrete Concrete
the task performed once a service is initialized"

LAYER
Svc Handler Connector
open()

 This pattern resolves the following forces

for network clients that use interfaces like SVC_HANDLER

sockets or TLI: A PEER_CONNECTOR

INIT
S
Connector
1. How to reuse active connection establishment code connect_svc_handler()

CONNECTION
PEER_STREAM
activate_svc_handler()
for each new service Svc Handlern handle_output()

LAYER
connect(sh, addr)
open() A
2. How to make the connection establishment code
portable across platforms that may contain sock- 1: connect_svc_handler
ets but not TLI, or vice versa (sh, addr);

activate_svc_handler
2:
3. How to enable exible policies for creation, con-
(sh);

nection establishment, and concurrency

REACTIVE
Event

LAYER
4. How to eciently establish connections with large Handler Reactor
number of peers or over a long delay path handle_output() n 1

125 126

Collaboration in the Connector

Collaboration in the Connector Pattern
Pattern con : peer_stream_ sh: reactor :
Client
Connector : SOCK Svc_Handler Reactor
Connector
connect(sh, addr)
CONNECTION

peer_stream_ FOREACH CONNECTION

INITIATION

con : sh: reactor :

CONNECTION INITIATION/

Client : SOCK Svc_Handler Reactor

SEVICE INITIALIZATION

PHASE

Connector INITIATE CONNECTION connect_svc_handler(sh, addr)

Connector
FOREACH CONNECTION connect(sh, addr) ASYNC CONNECT connect()
connect_svc_handler(sh, addr) INSERT IN REACTOR register_handler(con)
INITIATE CONNECTION
PHASE

SYNC CONNECT connect() handle_events()

START EVENT LOOP
activate_svc_handler(sh)
INITIALIZATION

ACTIVATE OBJECT
select()
open() FOREACH EVENT DO
SERVICE

handle_output()
PHASE

INSERT IN REACTOR register_handler(sh) CONNECTION COMPLETE

get_handle() activate_svc_handler(sh)
EXTRACT HANDLE
ACTIVATE OBJECT open()
handle_events() register_handler(sh)
PROCESSING

START EVENT LOOP

SERVICE

INSERT IN REACTOR
select()
PHASE

FOREACH EVENT DO get_handle()

handle_input() EXTRACT HANDLE
PROCESSING

DATA ARRIVES
SERVICE
PHASE

PROCESS DATA svc() handle_input()

DATA ARRIVES

PROCESS DATA svc()

 Synchronous mode
 Asynchronous mode
127 128
 Using the Connector Pattern for Connector Class Public Interface
the Gateway
 A reusable template factory class that es-
tablishes connections with clients
: Channel
: Channel
: Channel : Channel : Channel
: Channel template <class SVC_HANDLER, // Type of service
: Svc
: Svc : Svc : Svc class PEER_CONNECTOR> // Connection factory
Handler : Svc
Handler Handler Handler class Connector
Handler : Svc
Handler : public Service_Object
{
public:
: Connector ACTIVE
: Channel // Initiate connection to Peer.
CONNECTIONS
virtual int connect (SVC_HANDLER *svc_handler,
: Svc const PEER_CONNECTOR::PEER_ADDR &,
Handler Synch_Options &synch_options);
PENDING
CONNECTIONS
: Reactor
// Cancel a <svc_handler> that was
// started asynchronously.
virtual int cancel (SVC_HANDLER *svc_handler);

129 130

OO Design Interlude
 Q: What is the Synch Options class? Connector Class Protected
Interface
 A: This allows callers to dene the syn- protected:
chrony/asynchrony policies, e.g., // Demultiplexing hooks.
virtual int handle_output (HANDLE); // Success.
virtual int handle_input (HANDLE); // Failure.
class Synch_Options virtual int handle_timeout (Time_Value &, const void *);
{
// Options flags for controlling synchronization. // Create and cleanup asynchronous connections...
enum { virtual int create_svc_tuple (SVC_HANDLER *,
USE_REACTOR = 1, Synch_Options &);
USE_TIMEOUT = 2 virtual Svc_Tuple *cleanup_svc_tuple (HANDLE);
};
// Table that maps an I/O handle to a Svc_Tuple *.
Synch_Options (u_long options = 0, Map_Manager<HANDLE, Svc_Tuple *, Null_Mutex>
const Time_Value &timeout handler_map_;
= Time_Value::zero,
const void *arg = 0); // Factory that actively establishes connections.
PEER_CONNECTOR connector_;
// This is the default synchronous setting. };
static Synch_Options synch;
// This is the default asynchronous setting.
static Synch_Options asynch;
};

131 132
 Map Manager Class
OO Design Interlude
 Synchronization mechanisms are parameterized :::

 Q: \What is a good technique to imple-

menting a handler map?" template <class EXT_ID, class INT_ID, class LOCK>
class Map_Manager

{ e.g., to route messages or to map HANDLEs to {

public:
SVC HANDLERs bool bind (EXT_ID, INT_ID *);
bool unbind (EXT_ID);

bool find (EXT_ID ex, INT_ID &in) {

 A: Use a Map Manager collection class // Exception-safe code...
Read_Guard<LOCK> monitor (lock_);
{ ACE provides a Map Manager collection that as- // lock_.read_acquire ();
sociates external ids with internal ids, e.g., if (locate_entry (ex, in))
return true;
. External ids ! routing ID or HANDLE else
return false;
// lock_.release ();
. Internal ids ! set of Channel * or Svc Handler }

private:
{ Map Manager uses templates to enhance reuse LOCK lock_;
bool locate_entry (EXT_ID, INT_ID &);
// ...
};

133 134

Connector Class Implementation

// Shorthand names.
// Register a Svc_Handler that is in the
#define SH SVC_HANDLER
// process of connecting.
#define PC PEER_CONNECTOR
template <class SH, class PC> int
// Initiate connection using specified blocking semantics.
Connector<SH, PC>::create_svc_tuple
template <class SH, class PC> int
(SH *sh, Synch_Options &options)
Connector<SH, PC>::connect
{
(SH *sh, const PC::PEER_ADDR &r_addr, Synch_Options &options)
// Register for both "read" and "write" events.
{
Service_Config::reactor ()->register_handler
Time_Value *timeout = 0;
(sh->get_handle (),
int use_reactor = options[Synch_Options::USE_REACTOR];
Event_Handler::READ_MASK |
Event_Handler::WRITE_MASK);
if (use_reactor) timeout = Time_Value::zerop;
else
Svc_Tuple *st = new Svc_Tuple (sh, options.arg ());
timeout = options[Synch_Options::USE_TIMEOUT]
? (Time_Value *) &options.timeout () : 0;
if (options[Synch_Options::USE_TIMEOUT])
// Register timeout with Reactor.
// Use Peer_Connector factory to initiate connection.
int id = Service_Config::reactor ()->schedule_timer
if (connector_.connect (*sh, r_addr, timeout) == -1)
(this, (const void *) st,
{
options.timeout ());
// If the connection hasn't completed, then
st->id (id);
// register with the Reactor to call us back.
if (use_reactor && errno == EWOULDBLOCK)
// Map the HANDLE to the Svc_Handler.
create_svc_tuple (sh, options);
handler_map_.bind (sh->get_handle (), st);
} else
}
// Activate immediately if we are connected.
sh->open ((void *) this);
}

135 136
 // Finalize a successful connection (called by Reactor).

template <class SH, class PC> int

// Cleanup asynchronous connections...
template <class SH, class PC> Svc_Tuple *
Connector<SH, PC>::cleanup_svc_tuple (HANDLE h)
{ // Delegate control to the service handler.
Svc_Tuple *st;
// Locate the Svc_Tuple based on the handle;
handler_map_.find (h, st);
Connector<SH, PC>::handle_output (HANDLE h) {
Svc_Tuple *st = cleanup_svc_tuple (h);
// Transfer I/O handle to SVC_HANDLE *.
st->svc_handler ()->set_handle (h);
sh->open ((void *) this);
}
// Handle connection errors.

// Remove SH from Reactor's Timer_Queue.
Service_Config::reactor ()->cancel_timer (st->id ());
// Remove HANDLE from Reactor.
Service_Config::reactor ()->remove_handler (h,
Event_Handler::RWE_MASK | Event_Handler::DONT_CALL);
template <class SH, class PC> int
Connector<SH, PC>::handle_input (HANDLE h) {
Svc_Tuple *st = cleanup_svc_tuple (h);
}
// Handle connection timeouts.

// Remove HANDLE from the map. template <class SH, class PC> int
handler_map_.unbind (h); Connector<SH, PC>::handle_timeout
return st; (Time_Value &time, const void *arg) {
} Svc_Tuple *st = (Svc_Tuple *) arg;
st = cleanup_svc_tuple
(st->svc_handler ()->get_handle ());
// Forward "magic cookie"...
st->svc_handler ()->handle_timeout (tv, st->arg ());
}

137 138

Specializing Connector and Channel Inheritance Hierarchy

Svc Handler
APPLICATION-
 Producing an application that meets Gate- INDEPENDENT Svc
way requirements involves specializing ACE
components APPLICATION-
Handler
SPECIFIC
{ Connector ! Channel Connector Channel
peer_stream_
{ Svc Handler ! Channel ! Input Channel and
Output Channel

Output
 Note that these new classes selectively over- Channel Input
ride methods dened in the base classes msg_queue_
Channel
{ The Reactor automatically invokes these methods
in response to I/O, signal, and timer events

139 140
 Channel Class Public Interface OO Design Interlude
 Q: What is the MT SYNCH class and how
 Common methods and data for I/O Chan- does it work?
nels
 A: MT SYNCH provides a thread-safe syn-
// Determine the type of threading mechanism.
chronization policy for a particular instanti-
#if defined (ACE_USE_MT)
typedef MT_SYNCH SYNCH; ation of a Svc Handler
#else
typedef NULL_SYNCH SYNCH; { e.g., it ensures that any use of a Svc Handler's
#endif /* ACE_USE_MT */ Message Queue will be thread-safe
// This is the type of the Routing_Table. { Any Task that accesses shared state can use the
typedef Routing_Table <Peer_Addr, \traits" in the MT SYNCH
Routing_Entry,
SYNCH::MUTEX> ROUTING_TABLE; class MT_SYNCH { public:
typedef Mutex MUTEX;
class Channel typedef Condition<Mutex> CONDITION;
: public Svc_Handler<SOCK_Stream, SYNCH> };
{
public:
// Initialize the handler (called by Connector). { Contrast with NULL SYNCH
virtual int open (void * = 0);
class NULL_SYNCH { public:
// Bind addressing info to Router. typedef Null_Mutex MUTEX;
virtual int bind (const INET_Addr &, CONN_ID); typedef Null_Condition<Null_Mutex> CONDITION;
};

141 142

Detailed OO Architecture of the

Gateway
Channel Class Protected Interface : Output : Map : Routing : Input
Channel Manager Table Channel
: Message
: SOCK Queue : SOCK

 Common data for I/O Channels Stream

: Reactor
Stream

: Output
protected: : Input Channel
// Reconnect Channel if connection terminates. Channel : Connector : Acceptor : Message
: SOCK
virtual int handle_close (HANDLE, Reactor_Mask); : SOCK Stream
Queue
: SOCK : Map
Stream Manager : SOCK
Connector Acceptor
// Address of peer.
INET_Addr addr_;

INCOMING OUTGOING
// The assigned connection ID of this Channel. GATEWAY
MESSAGES
MESSAGES
CONN_ID id_; CONNECTION CONNECTION
}; REQUEST REQUEST

143 144
 Input Channel Interface Output Channel Interface
 Handle input processing and routing of mes-  Handle output processing of messages sent
sages from Peers to Peers
class Input_Channel : public Channel class Output_Channel : public Channel
{ {
public: public:
Input_Channel (void); Output_Channel (void);

protected: // Send a message to a Gateway (may be queued).

// Receive and process Peer messages. virtual int put (Message_Block *, Time_Value * = 0);
virtual int handle_input (HANDLE);
protected:
// Receive a message from a Peer. // Perform a non-blocking put().
virtual int recv_peer (Message_Block *&); int nonblk_put (Message_Block *mb);

// Action that routes a message from a Peer. // Finish sending a message when flow control abates.
int route_message (Message_Block *); virtual int handle_output (HANDLE);

// Keep track of message fragment. // Send a message to a Peer.

Message_Block *msg_frag_; virtual int send_peer (Message_Block *);
}; };

145 146

Channel Connector Class Channel Connector

Interface Implementation
 A Concrete factory class that behaves as
follows:  Initiate (or reinitiate) a connection to the
1. Establishes connections with Peers to produce Channels
Channel
int
2. Activates Channels, which then do the work Channel_Connector::initiate_connection (Channel *channel)
{
class Channel_Connector : public // Use asynchronous connections...
Connector <Channel, // Type of service if (connect (channel, channel->addr (),
SOCK_Connector> // Connection factory Synch_Options::asynch) == -1) {
{ if (errno == EWOULDBLOCK)
public: return -1; // We're connecting asynchronously.
// Initiate (or reinitiate) a connection on Channel. else
int initiate_connection (Channel *); // Failure, reschedule ourselves to try again later.
} Service_Config::reactor ()->schedule_timer
(channel, 0, channel->timeout ());
}
else

 Channel Connector also ensures reliability by // We're connected.

return 0;
restarting failed connections }

147 148
 Structure of the Router Pattern
The Router Pattern
Routing
Table
 Intent 1
find() Message
{ \Decouple multiple sources of input from multiple Queue
sources of output to prevent blocking"

ROUTING
n

LAYER
Input Output
Channel Channel
 The Router pattern resolves the following
forces for connection-oriented routers: handle_input() handle_output()
put()
{ How to prevent misbehaving connections from dis-
rupting the quality of service for well-behaved con-
nections

REACTIVE
LAYER
{ How to allow dierent concurrency strategies for Event
Input and Output Channels Handler 1 Reactor
n

149 150

Collaboration in the Router Collaboration in Single-threaded

Pattern Gateway Routing
: Routing : Output : Input reactor :
main() : Output
Table Channel Channel Reactor
: Routing Channel
handle_events()
START EVENT LOOP Table
PROCESSING

FOREACH EVENT DO select() : Message 5: send_peer(msg)

Queue
PHASE
INPUT

DATA EVENT handle_input()

ROUTE
RECV MSG ID
recv_peer() Subscriber
Set : Output
ROUTE MSG
find () 3: find() Channel
)
sg
SELECTION

FIND DESTINATIONS put()

m
t(
ROUTE

PHASE

send_peer() : Message
pu

SEND MSG
(QUEUE IF FLOW enqueue() Queue
4:

CONTROLLED)
schedule_wakeup() 6: put (msg) 7: send_peer(msg)
: Input 8: enqueue(msg)
PROCESSING

FLOW CONTROL
ABATES
handle_output() Channel 9: schedule_wakeup()
OUTPUT

---------------
PHASE

dequeue()
DEQUEUE AND SEND 10: handle_output()
MSG (REQUEUE IF 1: handle_input() 11: dequeue(msg)
put()
FLOW CONTROLLED) 2: recv_peer(msg) 12: send_peer(msg)

151 152
// Receive input message from Peer and route
// the message. // Route message from a Peer.

int int
Input_Channel::handle_input (HANDLE) Input_Channel::route_messages (Message_Block *route_addr)
{ {
Message_Block *route_addr = 0; // Determine destination address.
CONN_ID route_id = *(CONN_ID *) route_addr->rd_ptr ();
// Try to get the next message.
if ((n = recv_peer (route_addr)) <= 0) { const Message_Block *const data = route_addr->cont ();
if (errno == EWOULDBLOCK) return 0; Routing_Entry *re = 0;
else return n;
} // Determine route.
else Routing_Table::instance ()->find (route_id, re);
route_message (route_addr);
} // Initialize iterator over destination(s).
Set_Iterator<Channel *> si (re->destinations ());
// Send a message to a Peer (queue if necessary).
// Multicast message.
int for (Channel *out_ch;
Output_Channel::put (Message_Block *mb, Time_Value *) si.next (out_ch) != -1;
{ si.advance ()) {
if (msg_queue_->is_empty ()) Message_Block *newmsg = data->duplicate ();
// Try to send the message *without* blocking! if (out_ch->put (newmsg) == -1) // Drop message.
nonblk_put (mb); newmsg->release (); // Decrement reference count.
else }
// Messages are queued due to flow control. delete route_addr;
msg_queue_->enqueue_tail (mb, Time_Value::zerop); }
}

153 154

Message Queue and Peer Message

Message Block Object Diagram // unique connection id that denotes a Channel.
typedef short CONN_ID;

// Peer address is used to identify the

SYNCH
: Message // source/destination of a Peer message.
Queue : Message class Peer_Addr {
head_ Block public:
tail_ next_ CONN_ID conn_id_; // Unique connection id.
prev_ : Payload u_char logical_id_; // Logical ID.
cont_ u_char payload_; // Payload type.
};

: Message // Fixed sized header.

Block class Peer_Header { public: /* ... */ };
next_
prev_ : Payload // Variable-sized message (sdu_ may be
: Message
Block cont_ // between 0 and MAX_MSG_SIZE).
next_
prev_ : Message class Peer_Message {
cont_ Block public:
: Payload // The maximum size of a message.
: Payload enum { MAX_PAYLOAD_SIZE = 1024 };
Peer_Header header_; // Fixed-sized header portion.
char sdu_[MAX_PAYLOAD_SIZE]; // Message payload.
};

155 156

OO Design Interlude
 Q: What should happen if put() fails?
{ e.g., if a queue becomes full?
 A: The answer depends on whether the error
handling policy is dierent for each router
object or the same :::
{ Bridge pattern: give reasonable default, but allow
substitution via subclassing
 A related design issue deals with avoiding
output blocking if a Peer connection be-
comes
ow controlled
157
OO Design Interlude
 Q: How can a
ow controlled Output Channel // Perform a non-blocking put() of message MB.
know when to proceed again without polling
or blocking?
 A: Use the Event Handler::handle output no-
// Pseudo-code for receiving framed message
// (using non-blocking I/O).
int
Input_Channel::recv_peer (Message_Block *&route_addr)
{
if (msg_frag_ is empty) {
msg_frag_ = new Message_Block;
receive fixed-sized header into msg_frag_
if (errors occur)
cleanup
else
determine size of variable-sized msg_frag_
}
else
determine how much of msg_frag_ to skip
perform non-blocking recv of payload into msg_frag_
if (entire message is now received) {
route_addr = new Message_Block (sizeof (Peer_Addr),
msg_frag_)
Peer_Addr addr (id (), msg_frag_->routing_id_, 0);
route_addr->copy (&addr, sizeof (Peer_Addr));
return to caller and reset msg_frag_
}
else if (only part of message is received)
return errno = EWOULDBLOCK
else if (fatal error occurs)
cleanup
}
158
int Output_Channel::nonblk_put (Message_Block *mb)
{
// Try to send the message using non-blocking I/O
if (send_peer (mb) != -1
&& errno == EWOULDBLOCK)
tication scheme of the Reactor {
// Queue in *front* of the list to preserve order.
{ i.e., via the Reactor's methods schedule wakeup msg_queue_->enqueue_head (mb, Time_Value::zerop);
and cancel wakeup // Tell Reactor to call us back when we can send again.

Service_Config::reactor ()->schedule_wakeup
 This provides cooperative multi-tasking within (this, Event_Handler::WRITE_MASK);
a single thread of control }
}

{ The Reactor calls back to the handle output

method when the Channel is able to transmit
again

159 160
 // Finish sending a message when flow control
// Simple implementation... // conditions abate. This method is automatically
// called by the Reactor.
int
Output_Channel::send_peer (Message_Block *mb) int
{ Output_Channel::handle_output (HANDLE)
ssize_t n; {
size_t len = mb->length (); Message_Block *mb = 0;

// Try to send the message. // Take the first message off the queue.
n = peer ().send (mb->rd_ptr (), len); msg_queue_->dequeue_head
(mb, Time_Value::zerop);
if (n <= 0) if (nonblk_put (mb) != -1
return errno == EWOULDBLOCK ? 0 : n; || errno != EWOULDBLOCK) {
else if (n < len) // If we succeed in writing msg out completely
// Skip over the part we did send. // (and as a result there are no more msgs
mb->rd_ptr (n); // on the Message_Queue), then tell the Reactor
else /* if (n == length) */ { // not to notify us anymore.
delete mb; // Decrement reference count.
errno = 0; if (msg_queue_->is_empty ()
} Service_Config::reactor ()->cancel_wakeup
return n; (this, Event_Handler::WRITE_MASK);
} }
}

161 162

The Gateway Class

Connector Service Map Gateway Class Public Interface

Object Manager

APPLICATION-
 Since Gateway inherits from Service Object
INDEPENDENT it may be dynamically (re)congured into a
process at run-time
APPLICATION-
SPECIFIC
Channel Config // Parameterized by the type of I/O channels.
Connector Table template <class INPUT_CHANNEL, // Input policies
class OUTPUT_CHANNEL> // Output policies
SINGLETON class Gateway
Routing : public Service_Object
Table {
INPUT CHANNEL public:
OUTPUT CHANNEL SINGLETON

Gateway // Perform initialization.

virtual int init (int argc, char *argv[]);

// Perform termination.
virtual int fini (void);

 This class integrates other application-specic

and application-independent components
163 164
 // Convenient short-hands.
#define IC INPUT_CHANNEL
#define OC OUTPUT_CHANNEL

Gateway Class Private Interface // Pseudo-code for initializing the Gateway (called
// automatically on startup).
protected:
// Parse the channel table configuration file.
template <class IC, class OC>
int parse_cc_config_file (void);
Gateway<IC, OC>::init (int argc, char *argv[])
{
// Parse the routing table configuration file.
// Parse command-line arguments.
int parse_rt_config_file (void);
parse_args (argc, argv);
// Initiate connections to the Peers.
// Parse and build the connection configuration.
int initiate_connections (void);
parse_cc_config_file ();
// Table that maps Connection IDs to Channel *'s.
// Parse and build the routing table.
Map_Manager<CONN_ID, Channel *, Null_Mutex>
parse_rt_config_file ();
config_table_;
};
// Initiate connections with the Peers.
initiate_connections ();
return 0;
}

165 166

Conguration and Gateway Conguration Files

Routing  The Gateway decouples the connection topol-
ogy from the peer routing topology
{ The following cong le species the connection
PEER
1
PEER
2 topology among the Gateway and its Peers
# Conn ID Hostname Port Direction Max Retry
# ------- -------- ---- --------- ---------
1 peer1 10002 O 32
GATEWAY (30, 9) => 1 2 peer2 10002 I 32
(21,10) => 5 3 peer3 10002 O 32
(9,8) => 3 4 peer4 10002 I 32
5 peer5 10002 O 32
PEER PEER
5 3
{ The following cong le species the routing topol-
(12,13) => 1,3 ogy among the Gateway and its Peers
(13,8) => 5
# Conn ID Logical ID Payload Destinations
PEER # ------- ---------- ------- ------------
4 2 30 9 1
2 21 10 5
2 09 8 3
4 12 13 1,3
4 13 8 5

167 168
 // Parse the rt_config_file and
// build the routing table.
// Parse the cc_config_file and
// build the connection table.
template <class IC, class OC>
Gateway<IC, OC>::parse_rt_config_file (void)
template <class IC, class OC>
{
Gateway<IC, OC>::parse_cc_config_file (void)
RT_Entry entry;
{
rt_file.open (cc_filename);
CC_Entry entry;
cc_file.open (cc_filename);
// Example of the Builder Pattern.
// Example of the Builder Pattern.
while (cc_file.read_line (entry) {
Routing_Entry *re = new Routing_Entry;
while (cc_file.read_line (entry) {
Peer_Addr peer_addr (entry.conn_id, entry.logical_id_);
Channel *ch;
Set<Channel *> *channel_set = new Set<Channel *>;
// Locate/create routing table entry.
// Example of the Iterator pattern.
if (entry.direction_ == 'O')
foreach destination_id in entry.total_destinations_ {
ch = new OC;
Channel *ch;
else
if (config_table_.find (destination_id, ch);
ch = new IC;
channel_set->insert (ch);
}
// Set up the peer address.
INET_Addr addr (entry.port_, entry.host_);
// Attach set of destination channels to routing entry.
ch->bind (addr, entry.conn_id_);
re->destinations (channel_set);
ch->max_timeout (entry.max_retry_delay_);
config_table_.bind (entry.conn_id_, ch);
// Bind with routing table, keyed by peer address.
}
routing_table.bind (peer_addr, re);
}
}
}

169 170

Dynamically Conguring Services

// Initiate connections with the Peers.

int Gateway<IC, OC>::initiate_connections (void)

into an Application
{
// Example of the Iterator pattern.
Map_Iterator<CONN_ID, Channel *, Null_Mutex>  Main program is generic
cti (connection_table_);

// Example of the Service Configurator pattern.

// Iterate through connection table
// and initiate all channels.
int main (int argc, char *argv[])
{
for (const Map_Entry <CONN_ID, Channel *> *me = 0;
Service_Config daemon;
cti.next (me) != 0;
// Initialize the daemon and
cti.advance ()) {
// dynamically configure services.
Channel *channel = me->int_id_;
daemon.open (argc, argv);
// Initiate non-blocking connect.
// Run forever, performing configured services.
Channel_Connector::instance ()->
initiate_connection (channel);
daemon.run_reactor_event_loop ();
}
return 0;
/* NOTREACHED */
}
}

171 172
 Dynamic Linking a Gateway
Using the Service Congurator Service
Pattern for the Gateway
 Service conguration le
SERVICE : Thread % cat ./svc.conf
CONFIGURATOR
Gateway static Svc_Manager "-p 5150"
RUNTIME
: Reactive dynamic Gateway_Service Service_Object *
Gateway : Service Gateway.dll:make_Gateway () "-d"
SHARED
Object
: Service : Service OBJECTS
Repository Object
: Thread Pool  Application-specic factory function used to
: Service : Reactor
Gateway
dynamically link a service
Config : Service
Object // Dynamically linked factory function that allocates
// a new single-threaded Gateway object.

extern "C" Service_Object *make_Gateway (void);

Service_Object *
 Replace the single-threaded Gateway with a make_Gateway (void)
{
multi-threaded Gateway return new Gateway<Input_Channel, Output_Channel>;
// ACE automatically deletes memory.
}

173 174

Concurrency Strategies for

Patterns Using the Active Object Pattern
for the Multi-threaded Gateway
 The Acceptor and Connector patterns do
not constrain the concurrency strategies of REGISTERED
APPLICATION

: Output : Output OBJECTS : Input

a Svc Handler : Input
LEVEL

Channel Channel Channel : Input

Channel
4: send(msg) Channel
: Event : Event : Event
: Event
 There are three common choices: Handler Handler Handler : Event
Handler
2: recv(msg)
3: route(msg)
Handler

1. Run service in same thread of control 1: handle_input()

FRAMEWORK

2. Run service in a separate thread

LEVEL

:Timer : Handle
Table : Signal
Queue
Handlers
3. Run service in a separate process : Reactor
OS EVENT DEMULTIPLEXING INTERFACE
KERNEL

Observe how OO techniques push this de-

LEVEL


cision to the \edges" of the design
{ This greatly increases reuse, exibility, and perfor-
mance tuning

175 176
 Collaboration in the Active Half-Sync/Half-Async Pattern
Object-based Gateway Routing
 Intent
{ \Decouple synchronous I/O from asynchronous I/O
in a system to simplify concurrent programming
: Routing
Table
: Output
Channel
eort without degrading execution eciency"
5: send_peer(msg)
: Message
Queue
 This pattern resolves the following forces
ROUTE
ID
ACTIVE for concurrent communication systems:
Subscriber
3: find() Set
{ How to simplify programming for higher-level com-
4: put (msg) munication tasks
: Output
Channel . These are performed synchronously (via Active
: Input : Message
Objects)
Channel Queue
ACTIVE { How to ensure ecient lower-level I/O communi-
1: handle_input ()
cation tasks
5: send_peer(msg)
2: recv_peer(msg)
. These are performed asynchronously (via Reac-
tor)

177 178

Structure of the
Half-Sync/Half-Async Pattern Collaborations in the
Half-Sync/Half-Async Pattern
SYNC SYNC External Async Message Sync
SYNCHRONOUS
TASK LAYER

TASK 1
TASK 3 Event Source Task Queue Task
notification()
EXTERNAL EVENT
SYNC QUEUEING ASYNC
PHASE

read(msg)
RECV MSG
SYNC work()
TASK 2 PROCESS MSG

enqueue(msg)
PHASE PHASE
QUEUEING

ENQUEUE MSG
LAYER

1, 4: read(data) read(msg)
DEQUEUE MSG

work()
MESSAGE QUEUES EXECUTE TASK

3: enqueue(data)
ASYNCHRONOUS
TASK LAYER

ASYNC
TASK 2: interrupt

EXTERNAL
 This illustrates input processing (output pro-
EVENT SOURCES cessing is similar)

179 180
 Using the Half-Sync/Half-Async Class Diagram for Multi-Threaded
Pattern in the Gateway Gateway
Channel SOCK_Stream

APPLICATION-

COMPONENTS
QUEUEING SYNCHRONOUS

: Output : Output SOCK_Connector MT_Synch

TASK LAYER

1 n

SPECIFIC
Channel : Output Channel Channel Input
Channel Connector ACTIVATES /Thr_Output
Channels

CONNECTION-

COMPONENTS
SVC_HANDLER
1: dequeue(msg) PEER_STREAM

ORIENTED
PEER_CONNECTOR SYNCH
2: send(msg) Svc
Connector
LAYER

Handler

MESSAGE QUEUES
PEER PEER
2: recv(msg) CONNECTOR STREAM

3: get_route(msg)
ASYNCHRONOUS

COMPONENTS
FRAMEWORK
TASK LAYER

Stream
4: enqueue(msg) Connection

ACE
: Input IPC_SAP
: Input 1: dispatch() Service
Channel
: Input
Channel Configurator
Channel : Reactor Reactor
Concurrency
global

181 182

Thr Output Channel Class

Interface Thr Output Channel Class
Implementation
 New subclass of Channel uses the Active Ob-
ject pattern for the Output Channel
 The multi-threaded version of open is slightly
{ Uses multi-threading and synchronous I/O (rather dierent since it spawns a new thread to be-
than non-blocking I/O) to transmit message to
Peers come an active object!
{ Transparently improve performance on a multi-
processor platform and simplify design // Override definition in the Output_Channel class.

int
#define ACE_USE_MT
Thr_Output_Channel::open (void *)
#include "Channel.h"
{
// Become an active object by spawning a
class Thr_Output_Channel : public Output_Channel
// new thread to transmit messages to Peers.
{
public:
activate (THR_NEW_LWP | THR_DETACHED);
// Initialize the object and spawn a new thread.
}
virtual int open (void *);

// Send a message to a peer.

virtual int put (Message_Block *, Time_Value * = 0);

// Transmit peer messages within separate thread.

 activate is a pre-dened method on class
virtual int svc (void); Task
};

183 184
 Dynamic Linking a Gateway
// Queue up a message for transmission (must not block
Service
// since all Input_Channels are single-threaded).

int
Thr_Output_Channel::put (Message_Block *mb, Time_Value *)
 Service conguration le
{
// Perform non-blocking enqueue. % cat ./svc.conf
msg_queue_->enqueue_tail (mb, Time_Value::zerop); remove Gateway_Service
} dynamic Gateway_Service Service_Object *
thr_Gateway.dll:make_Gateway () "-d"
// Transmit messages to the peer (note simplification
// resulting from threads...)

int  Application-specic factory function used to

Thr_Output_Channel::svc (void)
{
dynamically link a service
Message_Block *mb = 0;
// Dynamically linked factory function that allocates
// Since this method runs in its own thread it // a new multi-threaded Gateway object.
// is OK to block on output.
extern "C" Service_Object *make_Gateway (void);
while (msg_queue_->dequeue_head (mb) != -1)
send_peer (mb); Service_Object *
make_Gateway (void)
return 0; {
} return new Gateway<Input_Channel, Thr_Output_Channel>;
// ACE automatically deletes memory.
}

185 186

Lessons Learned using OO Lessons Learned using OO

Design Patterns Frameworks
 Benets of patterns
{ Enable large-scale reuse of software architectures
{ Improve development team communication
{ Help transcend language-centric viewpoints
 Drawbacks of patterns
{ Do not lead to direct code reuse
{ Can be deceptively simple
ow of control for reactive dispatching is non-
{ Teams may suer from pattern overload
187
 Benets of frameworks
{ Enable direct reuse of code (cf patterns)
{ Facilitate larger amounts of reuse than stand-alone
functions or individual classes
 Drawbacks of frameworks
{ High initial learning curve
. Many classes, many levels of abstraction
{ The
intuitive
188
 Obtaining ACE
Lessons Learned using C++
 The ADAPTIVE Communication Environ-
 Benets of C++
{ Classes and namespaces modularize the system ar-
chitecture
{ Inheritance and dynamic binding decouple applica-
tion policies from reusable mechanisms
{ Parameterized types decouple the reliance on par-
ment (ACE) is an OO toolkit designed ac-
cording to key network programming pat-
terns
 All source code for ACE is freely available
{ Anonymously ftp to wuarchive.wustl.edu
ticular types of synchronization methods or net- { Transfer the les /languages/c++/ACE/*.gz and
work IPC interfaces gnu/ACE-documentation/*.gz

 Drawbacks of C++  Mailing lists

{ Many language features are not widely implemented * ace-users@cs.wustl.edu
* ace-users-request@cs.wustl.edu
* ace-announce@cs.wustl.edu
{ Development environments are primitive * ace-announce-request@cs.wustl.edu
{ Language has many dark corners and sharp edges  WWW URL
{ http://www.cs.wustl.edu/~schmidt/ACE.html
189 190

Patterns Literature
Relevant Conferences and
 Books Workshops
{ Gamma et al., \Design Patterns: Elements of
Reusable Object-Oriented Software" Addison-Wesley,
1994  Joint Pattern Languages of Programs Con-
ferences
{ Pattern Languages of Program Design series by
Addison-Wesley, 1995 and 1996 { 3rd PLoP
{ Siemens, Pattern-Oriented Software Architecture, September 4,6, 1996, Monticello, Illinois, USA
Wiley and Sons, 1996 .

{ 1st EuroPLoP
 Special Issues in Journals
. July 10,14, 1996, Kloster Irsee, Germany
{ Dec. '96 \Theory and Practice of Object Sys-
tems" (guest editor: Stephen P. Berczuk)
{ http://www.cs.wustl.edu/~schmidt/jointPLoP,96.html/
{ October '96 \Communications of the ACM" (guest
editors: Douglas C. Schmidt, Ralph Johnson, and
Mohamed Fayad)
 USENIX COOTS
 Magazines { June 17,21, 1996, Toronto, Canada
{ C++ Report and Journal of Object-Oriented Pro- { http://www.cs.wustl.edu/~schmidt/COOTS,96.html/
gramming, columns by Coplien, Vlissides, and Mar-
tin
191 192