Future Generation Computer Systems 22 (2006) 88–101

A middleware architecture to facilitate distributed programming

DAROC: Data-Activated Replicated Object Communications
Brian M. Stack, Gene Hsiao, Stephen F. Jenks∗
University of California, 444E Engineering Tower, Irvine, CA 92697-2625, USA

Received 6 December 2003; accepted 6 December 2003

Available online 25 December 2004

Abstract

Programming distributed computer systems is difficult because of complexities in addressing remote entities, message han-
dling, and program coupling. As systems grow, scalability becomes critical, as bottlenecks can serialize portions of the system.
When these distributed system aspects are exposed to programmers, code size and complexity grow, as does the fragility of the
system. This paper describes a distributed software architecture and middleware implementation that combines object-based
blackboard-style communications with data-driven and periodic application scheduling to greatly simplify distributed program-
ming while achieving scalable performance. Data-Activated Replication Object Communications (DAROC) allows programmers
to treat shared objects as local variables while providing implicit communications.
© 2004 Elsevier B.V. All rights reserved.

Keywords: Middleware; Distributed system; Data-activated; Blackboard

1. DAROC at a glance in DAROC, programmers are able to simply access

The Data-Activated Replication Object Communi-
cations (DAROC) approach empowers programmers
to be able to write components of distributed sys-
tems without being distributed systems experts, while
leaving the configuration of mission-critical systems
to architects educated in design complexities and pit-
falls of such systems. When writing an application
∗ Corresponding author. Tel.: +1 949 824 9072;
fax: +1 949 824 3203.
E-mail addresses: stackbr@hotmail.com (B.M. Stack),
gene@ucsd.edu (G. Hsiao), sjenks@uci.edu (S.F. Jenks).
the objects they want without having to be concerned
with the objects’ locations. Programmers have ac-
cess to both the data members and the member func-
tions associated with the object. For example, the
following simple code fragments show how to ac-
cess a member value in a temperature object in a
simulation:
Data member usage:
$temperature → current temp;
Member function usage:
$temperature → convert to celsius( );

0167-739X/$ – see front matter © 2004 Elsevier B.V. All rights reserved.
doi:10.1016/j.future.2003.12.025
 B.M. Stack et al. / Future Generation Computer Systems 22 (2006) 88–101
Logic usage:
if($temperature → currrent temp > 100)
$temperature → current temp = 100.
The ‘$’ symbol is used by the DAROC parser to
wrap objects in DAROC code. Code written to use
DAROC is normal C++ code that accesses local vari-
ables and distributed objects in similar ways. The ad-
dition of the ‘$’ notation tells the DAROC toolkit to
add code to manage the distributed nature of the object
without letting that distributed nature affect the pro-
grammer. The DAROC middleware handles distribu-
tion of objects and activation of subscribed applications
without any explicit communications or scheduling op-
erations in the application code.
2. Motivation
Distributed software is normally inherently more
complex than normal, sequential, monolithic software.
The complexity arises from local issues, like buffer and
concurrency management, to network-centric prob-
lems of naming, addressing, and message passing. Mid-
dleware solutions, like CORBA and others, have been
developed to abstract or hide some of the communica-
tion details, but the distributed nature of the system is
often still exposed and some complexity and overhead
are generally associated with such approaches. Devel-
opers must consider aspects such as communications
and scheduling in addition to the functional algorithm
of the code they are writing. For example, if a middle-
ware solution provides a remote procedure call (RPC)
mechanism, the developer must choose between block-
ing and non-blocking calls, and if non-blocking, worry
about buffer usage and synchronization upon comple-
tion. If, instead, the communication approach is mes-
sage based, the programmer must compose the mes-
sage, send it, receive the reply, and decode the reply
message. All of this work is in addition to the required
implementation of the function the program or module
is intended to perform.
The concepts behind DAROC arose from the goal
of eliminating the burden of communications and
scheduling from the application programmer while
maintaining performance and scalability of the sys-
tem. By giving programmers the view of communicat-
ing through shared objects, very natural interactions
between programs are facilitated. With the introduc-
89
tion of the single-writer rule and activation based on
data changes from data-flow and data-driven systems,
distributed systems issues such as race conditions and
explicit scheduling are eliminated. While the single-
writer rule for data objects (DO) may be seen by some
as a significant constraint on expressiveness, it elim-
inates behavior that limits scalability or would cause
significant confusion or overhead to manage.
Because the application programmers are simply
reading and writing data objects, their programs are
logically decoupled from one another through their
interfaces. If timing constraints require tight tempo-
ral coupling between two programs, the system engi-
neer can allocate both to the same node, where their
shared objects are implemented efficiently through
shared memory. If they can execute on separate nodes,
the DAROC runtime system handles object replication
transparently over the network. Neither case requires
any modification to the application program; thus, the
functional behavior of the programs in the system re-
mains the same.
DAROC borrows the central concept from the black-
board model, specifically that programs communicate
through shared data (objects, in this case) on a logically
shared medium. DAROC differs from conventional
blackboards because of the addition of data-driven
activation (or periodic activation, if appropriate to
the application). This eliminates any notion of and
need for a centralized controller and greatly enhances
the scalability of the system. By pushing scheduling
decisions out to the local processors and as close to the
distributed programs as feasible, there is no central bot-
tleneck. If programs are activated by changes in data,
they will activate on appropriate state changes. If they
are periodic, they see the current system state on the
blackboard when they run. This combination supports
the composition of significant distributed systems.
These specific goals help DAROC reach the overar-
ching goals below:
• Reduce software development costs, error-
proneness, and inflexibility associated with distri-
buted systems applications.
• Reduce code complexity so that the “average” pro-
grammer is able to build distributed systems.
• Provide a tool to better prepare students for the ever-
growing need in the job market for experienced dis-
tributed systems programmers.
90 B.M. Stack et al. / Future Generation Computer Systems 22 (2006) 88–101
This is not the first time a solution of this type
has been loosely based on the blackboard model
[14,15,21]. Most blackboard frameworks are not
geared around an object-oriented language, nor have
they been designed as a distributed middleware solu-
tion, with a few exceptions, and the majority of these
have complex control components. However, by laying
down a framework that is based on an object-oriented
language, uses data-driven activation to eliminate the
need of complex control structure, and taking the ini-
tial steps for a future distributed architecture, it does
become a unique approach that will provide valuable
information about the advantages, pitfalls, and overall
viability of such a solution.
The need for DAROC arose from the experience
of one of the authors while designing distributed
avionics architectures for next-generation aircraft. This
led to blackboard-style designs for avionics architec-
tures [9,11]. These prior systems did not have the
object-oriented nature of DAROC, nor did they ad-
dress shared-memory multiprocessor nodes within dis-
tributed systems. That author’s research into functional
language compilation and multithreaded architectures
[10] led to the functional programming and data-driven
activation paradigms of DAROC and the previous de-
signs. A non-distributed prototype implementation of
the DAROC middleware was shown in [22].
3. Background
Historically, the blackboard model arose from ab-
stracting features of the HEARSAY-II [7] speech un-
derstanding system, which was developed around 1971
and finished around 1976. The HEARSAY-II project
is considered the most influential on the development
of the blackboard model. HEARSAY-II was a project
under the DARPA Speech Understanding Project that
started in 1971. The Speech Understanding Project goal
was to design and implement a system that accepted
continuous speech from cooperative speakers that were
issuing commands in a quiet room with a good quality
microphone. HEARSAY-II, in particular, was able to
answer queries about computer science abstracts stored
in a database. HEARSAY-II is important because it is
noted as being the first true blackboard system.
The HASP project is another example of a black-
board system [18], which added even more credibil-
ity that the blackboard was a general problem-solving
method and a robust one. Shortly thereafter, many ap-
plications programs were developed in which the solu-
tion space was modeled around the blackboard model.
Since applications problems varied greatly along with
the interpretation of the blackboard model, the overall
design of each application differed widely.
TRICERO [12] is worthy of notice, since it was
an early attempt to apply the blackboard model in the
domain of distributed computing. It used the skeletal
AGE generalization. Its goal was to monitor a region of
airspace for various aircraft activities and understand
the signals and fuse together all of the information so
that it could draw a coherent picture. The problem was
partitioned into independent subproblems and various
blackboard systems perform the computations.
The backboard model of problem solving has not
undergone any substantial changes in the last three
decades. Nii [19] points out that the term “blackboard”
appeared in the Artificial Intelligence literature in 1962
when Newell was concerned with the problems of
programs organized along a generate-and-test search
model. Such programs were restricted in their data ac-
cess and had inflexible control posed by traditional pro-
gram structures. Newell realized that maintaining the
isolation of specific routines, but allowing all the sub-
routines to make use of a common data structure, could
alleviate the difficulties. Newell used the blackboard
metaphor to describe such a system as follows.
Metaphorically, we can think of a set of workers,
all looking at the same blackboard: each is able to read
everything that is on it, and to judge when he has some-
thing worthwhile to add to it. This conception is just
that of Selfridge’s Pandemonium [20]: a set of demons,
each independently looking at the total situation and
shrieking in proportion to what they see that fits their
natures [17].
A significant number of non-trivial applications
and frameworks have been built around the black-
board model. Some projects of note are the PROTEAN
project, which is a knowledge-based system that uses
the framework provided by BB1 [6–8]. It derives three-
dimensional conformations of proteins in solution from
empirical constraints derived from magnetic resonance
data. General BlackBoard (GBB) [4,5] is a generic de-
velopment system that provides an efficient and well-
structured blackboard that can be made available to a
“blackboard administrator” for the implementation and
maintenance of high-performance applications.
 B.M. Stack et al. / Future Generation Computer Systems 22 (2006) 88–101
One hot topic within the IT community that is ag-
gressively being pursued is that of audio mining via
audio search engines [16]. These speech engines offer
speech processing and search technologies in a single
package. The search engine usually creates an index
that includes a time and date stamp for each spoken
word. The search engine then uses the index to allow
rapid identification and playback of the requested mes-
sage. Meta-tags may also be placed on the data to iden-
tify the speaker, subject, or tone of the messages. This
new software breakthrough is being touted for indexing
uses ranging from corporate meetings to training videos
and even for help desk conversations. Companies such
as ScanSoft in Boston, Fast-Talk Communications in
Atlanta, and BBN Technologies in Cambridge, MA,
are all major players in this market who have an ap-
plication either built or influenced by the blackboard
model and are currently penetrating the commercial
marketplace.
3.1. The blackboard as the style of choice
The blackboard style’s simplicity is a big advantage
over other widely accepted architectural styles. The
blackboard style consists of one connector that all com-
ponents use, which of course is the blackboard itself.
Other blackboard advantages are flexibility and ability
to evolve, because new types of components can be eas-
ily written. A system based on CORBA [3] or JINI [2],
on the other hand, uses a client/server style that requires
programmers to design unique interfaces for every new
component that is generated. Since the DAROC black-
board style consists of only the blackboard interface
and very simple scheduling mechanisms, components
are simple to compose and integrate.
The blackboard style naturally lends itself to concur-
rency. Multiple components can read and write black-
board data and perform their assigned computations
concurrently, whereas styles such as pipe and filter or
layered systems are often serial in nature. These styles
consume their inputs, perform their computation, and
then send their results to the next component or layer.
Also, pipe and filter and layered system styles are not
nearly as flexible as the blackboard style and may have
scalability issues associated with them.
The DAROC architecture was partially influenced
by the object-oriented approach, in that DAROC data
objects consist of data and associated operations that
91
the functional elements (FEs) can exploit in an object-
oriented fashion. However, with DAROC, functional
elements do not know anything about the identities of
other functional elements and do not have to. There is
no such thing as a functional element communicating
directly with another functional element. The inverse
is the object-oriented style where objects must know
each other’s identities in order to take advantage of
the objects’ data and operations. The implicit invoca-
tion also has its influence in DAROC. DAROC uses
a data-activation approach to activate functional ele-
ments, which cause functional elements to be implic-
itly activated, based on changes to data object states
or periodic changes. Like implicit invocation, DAROC
components can be very heterogeneous and lend them-
selves to reusability in a variety of contexts. However,
DAROC, unlike other pure implicit invocation styles,
has knowledge of what components will respond to an
event, which in DAROC refers to updates to a data ob-
ject. DAROC uses subscription lists that are used to
link functional element to data objects.
Therefore, while DAROC is primarily based around
the blackboard style, in reality, like the vast majority of
systems built, it is heterogeneous in nature, combining
some parts of the blackboard, implicit invocation, and
object-oriented styles as part of its architecture.
4. DAROC as a middleware solution
DAROC exposes two architectural components to
programmers: functional elements are active and per-
form computations and analyze the system state, while
data objects that make up the communication method
between FEs and notionally reside on the black-
board. Unlike functional elements, data objects are
passive and do not perform computations. DOs are
strongly typed data objects, ostensibly defined with an
architecture- and programming language-independent
description, but for this prototype, normal C++ objects
are used. The prototype requires that DOs are contigu-
ous data structures (no pointers, for example) for sim-
plicity, because of the difficultly in transferring linked
objects between different address spaces. Eventually,
with the addition of packing methods, complex non-
contiguous objects can be supported.
Functional elements are the active entities in a
DAROC system and compose the distributed applica-
92 B.M. Stack et al. / Future Generation Computer Systems 22 (2006) 88–101
tion software. The scheduling of FEs is managed by
the middleware, and FEs are not designed to be contin-
uously running programs, but programs that are acti-
vated when needed, at which point, they perform their
function and then deactivate until the next time they are
needed. There is no limit to the size and complexity of
an FE. FEs can be as small as a simple data transfor-
mation, as large as a compiler, as complex as a target
tracking correlation filter or a graphics engine to draw
system status displays. Details of FE activation are pro-
vided below.
FEs get their name because they are intended
to be functional (in the mathematical sense) pro-
gram components. This means that FEs should com-
pute their results based on their inputs (DOs on the
blackboard) and produce results to be stored in DOs
on the blackboard. While it is not enforced, ideal
DOs should not store state internally, but should
store state in DOs on the blackboard. These guide-
lines provide some very valuable features to DAROC
systems:
1. Since the system state is stored on the blackboard
and FEs do not store internal state, debugging is sim-
plified. Developers can monitor and record the sys-
tem state unobtrusively by retrieving DOs of interest
from the blackboard. In addition, the FE should re-
turn identical results each time it is activated with a
specific set of inputs. Therefore, the functional cor-
rectness of the FE can be debugged independently
of the running system using test generators to stub
out the rest of the system.
2. Fault detection and recovery can also be facilitated
by taking periodic snapshots of the blackboard.
In non-real-time simulations, these snapshots can
be used to restart a system after node failures. In
embedded systems experiencing software failures,
these snapshots can be used to replay the state of
the system in the lab to help maintainers duplicate,
identify, isolate, and repair the faulty code.
3. If the DAROC middleware enforces replication of
all DOs on more than one node, recovery from a
node failure can be performed transparently. The
FEs that were on the failed node can be restarted
on another node, and using their most recent state
and input DOs from the blackboard can achieve the
proper results without any recovery code written
by the FE programmer. This is compelling because
error recovery code is fraught with potential errors
and is often difficult to test.
With the simplicity provided by local variable-
style access to transparently distributed DOs and
the functional nature of FEs, DAROC’s features
appeal to application programmers, just as the
above distributed system features appeal to system
architects.
4.1. DAROC’s distributed object space
While DOs appear to be local objects to the code
in an FE, DOs are uniquely identified by name in the
global system space. In the current prototype, DOs are
referenced using a single-level global naming scheme,
but DAROC’s goal is to utilize a robust hierarchical
one. The goal is to combine a set of FEs along with the
DOs they advertise as external interfaces and the inter-
nal DOs used to store state and communicate between
the FEs as a modular component that can be reused
between projects and simply inserted into systems as
needed. The blackboard does not distinguish between
DOs used within a component group and DO used to
communicate between components. Instead, the nam-
ing scheme separates the two. Since DOs are referenced
by name when they are created or opened, FEs from one
component group will tend to only access DOs that are
explicitly defined as external interfaces by other com-
ponents. At this moment, there is nothing to prevent
an FE from opening an “internal” DO from another
component, but internal interfaces may change more
frequently and with less documentation than external
ones, so doing so may invite trouble. Safeguards to pre-
vent unauthorized access to objects may be added in the
future if needed.
The distributed object space of DAROC allows DOs
to be read by any number of FEs, but each DO can only
be written by a single FE. This is known in DAROC
as the single-writer rule. By using a single-writer
rule as part of a programmer’s implementation of the
distributed system, we are able to easily eliminate race
conditions, ambiguity problems, and complex locking
mechanisms found in other distributed solutions
that allow shared-write access to data. While the
single-write rule initially seems like a significant
restriction to application programmers, we found that
programmers quickly adapt because of the simplicity of
 B.M. Stack et al. / Future Generation Computer Systems 22 (2006) 88–101
the rule. In addition, almost any situations that “need”
multiple writers to a single shared object can use a mod-
erator object to accept inputs from the multiple writer
FEs and then pass the appropriate value on to the desti-
nation DO. This approach is very scalable and does not
require global locking or other expensive synchroni-
zation.
DAROC data objects can be of arbitrarily complex
data types. All the simple data types are currently sup-
ported such as integers, doubles, and characters, as
are structured abstract data types, arrays, and combi-
nations thereof. The only limitation currently is that
all data objects that the user creates must be contigu-
ous storage type objects. At this point, linked lists,
trees, and other non-contiguous data types are not sup-
ported. A wrapper method to pack and unpack these
object types could be added to provide support in future
revisions.
DOs are referenced within FE code using normal
object pointers. The DAROC middleware creates and
manages the space for an FE’s DOs, so the FE can
simply access them as local objects. DAROC guaran-
tees the consistency of all DOs in use by an FE while
the FE is in its execution cycle (see below). When the
FE is not executing, DOs written by other FEs are up-
dated atomically, so no FE sees a partial update to
a DO. In addition, different FEs may have be using
different instances of common DOs. For example, if
one FE has a long execution cycle, it will use the in-
stances of its DOs that were current when its execu-
tion cycle began. Other FEs that start later may see
more recent instances of those same DOs. When an
FE completes an execution cycle, DAROC automati-
cally provides the current instance of DOs written by
that FE to other FEs that want them. By guarantee-
ing consistency during execution, DAROC provides a
simple, natural, expected behavior model to application
programmers.
The transfer of DOs between nodes is handled au-
tomatically by the DAROC middleware. Application
FEs are not aware of whether the source of any DO
they use is local or remote, so DAROC provides lo-
cation transparency. Since all communication is done
through named data objects, the distributed nature of
the system is hidden from application programmers.
Systems engineers can assign FEs to the same node
for performance or other considerations, but the code
does not change and the logical behavior remains
93
consistent no matter the placement. Section 4.3 de-
scribes the distributed object transfer mechanism in
detail.
4.2. DAROC’s FE Activation Cycle
A significant design constraint of DAROC is to
make scheduling of FEs both simple and scalable. This
is accomplished through activation when input data
changes (data-activation) or as time changes (periodic
activation). Many early systems built using the black-
board model used a very complex control component
for scheduling and activating the knowledge sources,
as they where called, which performed a computation
or analyzed the state of the blackboard. This added
control mechanism made blackboard implementations
complicated and, sometimes, fragile, because the con-
troller needed to understand the behavior of the system
in order to schedule it. This also limited scalability by
making the controller a bottleneck. In DAROC, activa-
tion of FEs is the result of changes to input data objects
specified by the application programmer. DAROC also
allows for the activation of functional elements on a
periodic basis. Since DAROC guarantees that all data
object updates are atomic from the consumer’s view-
point, once a functional element is activated, the data
object it uses will remain unchanged and accessible to
it. The functional element performs its work, computes
its results, and finishes and waits for the next time it is
activated.
Every functional element is created as a single
POSIX thread, which controls the functional element
payload. The functional element consists of wrapper
code generated by the DAROC toolset to manage DOs
and the application function written by the program-
mer to perform the work of the FE. Fig. 1 shows a
typical Activation Cycle for any generic functional
element.
Data objects are read and written by the FEs in the
system as often as needed. A handler associated with
each FE checks the specified set of data objects needed
by the FE to perform its computation. When the han-
dler notices a new instance of any of the DOs in its
set, it enters a verification phase. In this phase, the FE
handler first determines which data object has changed,
and then checks if any other DOs in its set have been
updated since the last time it activated the FE’s appli-
cation function. The normal activation condition in the
94 B.M. Stack et al. / Future Generation Computer Systems 22 (2006) 88–101

Fig. 1. FE Activation Cycle state diagram.

prototype is the “AND” condition, so if all DOs in the 4.3. Distributed object transfer
handler’s set have changed, the FE handler will pass
control to the application function that performs the Location-transparent implicit DO transfer between
work of the FE. While the AND activation condition is distributed system nodes is provided by DAROC.
the default, OR conditions or more complex Boolean A Name Server, written in Python, allows the FE
or state-machine operations are just as easily possi- handlers to look up the “home node” of any data
ble (complex conditions were not implemented for this object. The handlers can then query the home node for
prototype to avoid the complexity of how to specify each DO they need to determine the status of the DO.
them in a language-independent manner). For example, Fig. 2a shows how an FE handler looks up the source
the user may want to activate the FE every third update
to an input DO; rather than embedding that logic into
the application, it is better to specify it as an activation
condition in order to keep the functionality separate
from scheduling concerns. If the activation criteria are
not met, the handler will wait until it detects further
change in its DO set.
Inside the user application function, the user code
operates on the latest instances of the DOs in its com-
putational set. In the current prototype, these instances
are local copies made from the writer FEs’ original
objects. The handler stores these copies in the buffer
labeled FEBAR in Fig. 1 right before the application
function is started. Some of these copies are not really
needed for infrequently modified objects, so copy-on-
write and/or multiple buffer management with usage
counting will improve the efficiency in future revisions.
Once the application function finishes its computation,
control returns to the FE handler, which propagates up-
Fig. 2. Distributed object: (a) check in/lookup and (b) transfer re-
dated DO instances to the blackboard. quest.
 B.M. Stack et al. / Future Generation Computer Systems 22 (2006) 88–101
information for a DO from the Name Server. Fig. 2b
shows an FE handler requesting an updated DO
instance. The response message will either contain the
new instance or notification that the DO has not been
updated.
Because current implementation of DAROC is an
experimental prototype, we are still evaluating the best
update method to use for DO transfers. As of this writ-
ing, a “pull” approach is used, where the FE handlers
contact home nodes of DOs to determine if they have
been updated. The DAROC middleware on those nodes
will then either respond with the updated DO instance
or notification that no change has occurred since the last
time the FE received the DO (all instances are times-
tamped or otherwise uniquely identified when written
to allow such checks). This approach is bandwidth con-
servative, because it prevents unnecessary DO trans-
fers (to FEs that are not ready for the update), but it
also has some overhead and latency due to the queries.
We are evaluating a “push” approach where updated
DO instances are automatically transferred to nodes
containing FEs that use those DOs. This will enhance
scalability by making all update checks local, but may
increase network utilization due to excessive updates to
rapidly changing DOs. We are investigating throttling
approaches based on the activation patterns of FEs that
use such DOs, though fault tolerance may be hurt by
this (see below).
The automatic transfer of DO instances to nodes
that use them means that the “current” instance of a
DO may reside on more than one node. This natural
replication mechanism provides a simple redundancy
approach that can be used to recover from node failures.
If the replication is enforced so that all DOs exist on at
least two nodes, then recovery from failure of one of
the nodes can occur without any explicit recovery pro-
gramming from the user (assuming the FEs are indeed
functional and store any state as DOs on the black-
board). The FEs from the failed node can be restarted
on other nodes (either cold, warm, or hot spares), while
their input (and state) DOs can be retrieved from wher-
ever they reside. The middleware can then reassign the
identify of the writer node of the DOs written by the
restarted FEs to the new node(s) where they reside. This
functionality has not been completely implemented in
the current DAROC prototype, but many pieces are
in place. The forced replication is not done because it
likely needs the “push” transfer model described above.
95
4.4. The DAROC toolkit
The DAROC middleware provides network trans-
parency and a programming paradigm in which all
accesses to data objects appear as local accesses.
However, in order to facilitate the initial design
goal of simplifying distributed programming so that
the “average” programmer can easily and rapidly
begin writing distributed applications, a set of tools
will simplify code development. The tools will aid
application developers by automatically generating
wrapper/handler code to manage DO transfers and FE
activation. Other tools build FE prototype code and
verify the consistency of the blackboard.
The current toolkit contains three main components.
The first component is the DAROC parser. All FE ap-
plication source code is first run through this parser be-
fore it is compiled. The parser automatically wraps the
referenced data objects with the appropriate DAROC
code and casts the data objects to their corresponding
data types. This parser interprets names prefixed with
a ‘$’ symbol, as shown in a previous example, as DOs
to be managed by the middleware. It then matches the
DO name with the used specified class/type and gen-
erates the code to support DO access and transfer as
well as FE activation. The long-term goal of this por-
tion of the toolkit is to make DO type definitions use
an architecture- and language-independent specifica-
tion in order to support heterogeneous nodes. The tool
will be able to generate code to translate between for-
mats and orderings within the system.
The second component is the single-writer rule
checker. The single-writer rule is a main part of the
initial DAROC architecture. The single-writer rule
checker will examine user application code and warn
the programmer of any code segments, which could
possibly violate the single-writer rule. For the proto-
type, we chose to implement this rule checker as part of
the toolset rather than part of the middleware to reduce
overhead. We will re-evaluate this decision as needs
arise, but certainly the Name Server component could
be used to perform this check at run time.
The third toolkit component is the DAROC nugget
code creator. All DOs and FEs the user creates can
be derived from a standard code template generated
by this tool. That is to say, simple data objects are
created by initializing there private members and
calling DAROC’s data object constructor and simple
96 B.M. Stack et al. / Future Generation Computer Systems 22 (2006) 88–101
Fig. 3. DAROC Tool Kit Functional Element Code Nugget Form.
functional elements are created with a similar method
with the addition of the user FE application function
which is added. Both data objects and functional
elements also have their standard lists of which entities
they need to communicate with. The DAROC nugget
code generator will allow programmers in practical
terms to fill out a “form” for a simple data object or
functional element as shown in Fig. 3. For example,
in the functional element form, the programmer will
enter things like the name of the FE, its type, does it
have a periodic activation or is it based on specific data
objects. Also the programmer will enter how many
data objects it can read or write too and the names
of these objects, respectively. They can then simply
click a button and have the code generated for them,
with no missing commas, parentheses, or other similar
syntactical programming errors that so often occur.
The code nugget will be generated in a separate file,
so it will be up to the programmer to inject the code
pieces into the appropriate segments of the applica-
tion.
The DAROC nugget code generator will be valid
for simple functional elements and data objects, such
as ones with small subscription lists and those without
special user-defined member functions. Later improve-
ments to the component should make it more scalable to
a wider range of data objects and functional elements.
5. Results and analysis
DAROC has the seemingly competing goals of mak-
ing distributed application programming easier while
maintaining scalability and reasonable performance.
As the results below show, DAROC simplifies pro-
gramming by requiring much less application code than
competitive approaches. Performance test results also
show that even in its unoptimized state, the current
 B.M. Stack et al. / Future Generation Computer Systems 22 (2006) 88–101
prototype supports thousands of activations/object
transfers per second using shared memory.
To get a baseline evaluation of how DAROC can
make distributed programming easier, we examined
how JINI and JavaSpaces [1] from SUN Microsystems
expedite distributed programming for the programmer.
Fig. 4 shows a comparison of how JINI, JavaSpaces,
and DAROC stacked up in the number of lines of code
and number of characters to implement components of
a smart refrigerator demo that was written in DAROC.
The smart refrigerator simulation was written using
DAROC in order to give a visual projection of how
functional elements work and how they interact with
their respective data objects. A week of random re-
frigerator use was simulated. An inventory check, a
shopping list, numerous food scales, along with vari-
ous temperature and mechanical controllers, were also
simulated. As Fig. 4 illustrates, JINI is a very heavy so-
lution and takes considerably more lines of code then
either JavaSpaces or DAROC. JavaSpaces has a 10%
overhead compared to DAROC when looking only at
the number of lines of code.
However, when you examine the complexity of each
line of code in terms of the number of characters, which
we make the reasonable assumption that if a line of
code has half as many characters as another line of
code, then it is also proportionally less complex. It is
observed that DAROC overall takes about 22% less
characters to produce the same results. At face value,
this does not seem all that striking.
But if we were coding a realistic real world appli-
cation with tens-of-thousands of lines of code made up
hundreds-of-thousands of characters, then that 22% re-
Fig. 4. Analysis of three data objects and three functional elements from the smart appliance demo.
97
duction would vastly increase the speed at which the
application could be up and running.
5.1. Single node throughput measurement
A small system of two functional elements and
four data objects were used to measure the through-
put in terms of object update operations followed by
FE activations. These measurements were performed
on an Intel 810 Chipset, 1.1 GHz Celeron processor
and 384 MB of memory running Mandrake Linux 9.0.
Roughly two dozen simulations were performed, which
where based on average data object sizes ranging from
less than 500 bytes up to 1,000,000 bytes and total
activations which ranged from 500 to 1,000,000 (the
total number of activations for the system is achieved
by the addition of the two functional elements total
activations). Fig. 5 shows the results of these exper-
iments. It is important to keep in mind these num-
bers are strongly tied to the underlying hardware of
the system, particularly the CPU speed and memory
bandwidth.
In Fig. 5, an analysis of how long it takes for the
two functional elements using various average sized
data objects to perform the specified number of activa-
tions is shown. The figure shows that if the average data
object is under 10 KB, then DAROC can perform hun-
dreds to tens-of-thousands of activations in less than
1 s. Hundreds-of-thousands to millions of activations
can be performed in a matter of seconds as long as
the average data object size stays under 1 KB. When
more than hundreds-of-thousands of activations occur
or data objects greater than 10 KB are simulated, then
98 B.M. Stack et al. / Future Generation Computer Systems 22 (2006) 88–101
Fig. 5. Single node activation performance.
a linear increase in processing time occurs due to the
PC’s hardware limitations.
5.2. Distributed transfers
The DAROC prototype supports distributed DO
transfers correctly, but because of the way it is imple-
mented, it is difficult to get accurate measurements of
activations run per second or transfers per second. The
issues do not affect functionality, but currently, at least
one of the FEs that writes a DO drive the data-driven
FEs in the system must be periodic. Thus, a full-speed
feedback loop-style pair of FEs, which could be used
to measure activations per second, will not work with
the current code. While possible fixes to this issue are
being evaluated, we will analyze the process of activa-
tion and transfer in a simple FE pair to show that the
DAROC architecture is not, by nature, significantly less
efficient even than hand-coded message passing.
Fig. 6 shows a pair of interacting FEs that may run on
different nodes or the same node without any change to
the application code. Both FEs use both the “hello” DO
and the “world” DO as input. When both have changed,
each FE prints “helloworld” to the screen in its user-
written application function. FEhello writes the “hello”
DO and FEworld writes the “world” DO. The diagram
shows that all the activities involved with requesting
updated DOs and sending and receiving them are han-
dled by the DAROC-supplied handler rather than by
the application function. While some of the code in the
handlers is more general than code written for a single,
specific application, the overhead is very small, partic-
ularly on modern processors. The message traffic in this
simple case is the same request/response message pairs
as would be needed by a hand-coded communication
application. DAROC is intended to have similar perfor-
mance to hand-coded alternatives, but with less vari-
ability, because DAROC uses a consistent approach to
object transfers. On the other hand, DAROC (and other
general middleware solutions) may not take advantage
of application-specific optimizations that a clever pro-
grammer could fathom. This seems a fair trade, given
the complexity and fragility of distributed system pro-
gramming.
5.3. Function point analysis
Function points are a unit of measure for software
much like an hour is a measurement of time, miles are to
measuring distance, or Celsius is to measuring temper-
ature. Function points are an ordinal measure much like
other measures such as kilometers, Fahrenheit, hours,
etc. [13].
Function points measure software by quantifying its
functionality provided to the user based primarily on
the logical design. Frequently, the term ‘end user’ or
 B.M. Stack et al. / Future Generation Computer Systems 22 (2006) 88–101
Fig. 6. Two FEs interact through DO transfer.
‘user’ is used without specifying what is meant. In this
case, the user is a sophisticated user that is someone
that would understand the system from a functional per-
spective who is more than likely someone that would
provide requirements or does acceptance testing [13].
There are a variety of different methods used to
count function points, but this analysis is based upon
those rules developed by Alan Albrecht and later re-
vised by the International Function Point User Group
(IFPUG). There are varying types of function point
analysis that can be performed depending on the anal-
ysis you want to extract and the stage of the project.
The types of function point calculations that exist are:
• development project function point;
• enhancement project function point;
• application after enhancement project;
• application function point baseline count.
The analysis that is performed on DAROC and
JavaSpaces is an application function point baseline
count (AFPB). The AFPB = ADD × VAF, where ADD
is the unadjusted function point (UAF) count of those
functions that were added by the enhancements and
VAF is the value adjusted factor. Therefore, our calcu-
lation is AFPB = UAF × VAF. The analysis is based on
code from the smart appliance demo. The DOs and FEs
for the demo were analyzed, but the graphical front end,
which was built with the GTK+ graphics libraries, was
not analyzed. VAF is based on 14 general system char-
acteristics. The VAF measurements are becoming less
frequently used by organizations involved with func-
tion point analysis. Instead, many are using the UAF
99
with other metrics. The VAF for a DAROC and JavaS-
paces will be the same, since the general functionality
of both is similar. In addition, the analysis only takes
a subset of the functionality of the system for its com-
parison. Therefore, the VAF will not be used in the
analysis. So we are left with an analysis calculation of
AFPB = UAF.
A UAF calculation is based on fives types of com-
ponents. These components are:
• External Inputs (EI)—an elementary process in
which data cross the boundary from outside to in-
side;
• External Outputs (EO)—an elementary process in
which derived data pass across the boundary from
inside to outside;
• External Inquiry (EQ)—an elementary process with
both input and output components that result in data
retrieval from one or more Internal Logical Files and
External Interface Files;
• Internal Logical Files (ILF)—a user identifiable
group of logically related data that resides entirely
within the application boundary and is maintained
through External Input;
• External Interface Files (EIF)—a user identifiable
group of logically related data that is used for refer-
ence purposes only.
All of these components are rated as being low, av-
erage, or high; this rating is based on the number of
data element types (DET) that they reference. A DET
is a unique user recognizable non-recursive field. Since
the data sets in the smart appliance demo are small, all
100 B.M. Stack et al. / Future Generation Computer Systems 22 (2006) 88–101
Table 1
Function points complexity comparison
Type of component JavaSpaces DAROC
External Input 10 × 3 = 30 3×3=9
External Output 8 × 4 = 32 4 × 4 = 16
External Inquiries 20 × 3 = 60 17 × 3 = 51
Internal Logical Files 15 × 7 = 105 15 × 7 = 105
External Logical Files 0×5=0 0×5=0
Total UAF 227 181
components are rated at the low complexity level. A
DET can be viewed as the data in a data object in
DAROC or entry in JavaSpaces. Table 1 shows the com-
putation of component complexity for JavaSpaces and
DAROC. The numbers in bold are static and are the
set values when performing an analysis of a low com-
plexity level component. The italic numbers are the
total number of instances that type of component oc-
curred in the respective implementation of the various
data objects and functional elements in either DAROC
or JavaSpaces. This analysis reveals that DAROC has
a reduction in the number of explicit inputs, outputs,
inquires, and Internal Logical File components the pro-
grammer uses by a factor of 25.4%.
6. Continuing research
Future revisions of DAROC over the next few years
will continue to add system improvements such as com-
plex data objects, grid support, dynamic configuration
capabilities, and legacy encapsulation. The short-term
goal of DAROC is to provide an educational tool to
better teach distributed systems. The long-term goal is
for DAROC to address problems such as distributed
simulations and battle management scenarios.
7. Conclusion
Current distributed programming paradigms force
programmers to write communication and scheduling
code into their application code. This leads to unread-
able code, complex and hidden coupling and depen-
dences between programs, and makes the programs
more difficult to develop and maintain. We have shown
that the DAROC data-activated middleware approach
makes distributed application programming easier for
people who are not distributed systems experts. Be-
cause DAROC provides location transparent access
to “shared” data objects residing on a distributed vir-
tual blackboard, no explicit communications code is
required in applications. Because applications can be
activated by changes in the data objects they use or pe-
riodically, scheduling is likewise eliminated from the
applications. Analysis of programs written for DAROC
demonstrates a reduction in lines of code by 10–40%
and a reduction in code complexity based on function
point analysis by 25% when compared to other two
mainstream distributed programming environments.
We have shown that the middleware can automati-
cally transfer data objects between the nodes containing
the programs that write the objects and nodes running
programs that use the objects. This automatic replica-
tion, combined with simple software implementation
rules, lends itself to a fault tolerance model in which
programs from failed nodes can be restarted from their
replicated state objects. Our data transfer and activa-
tion experiments show that, while there are some inef-
ficiencies in the current prototype, single node perfor-
mance is good and multi-node performance is reason-
able. The middleware is doing network operations like
those that explicitly coded distributed software would
do, but it is doing it in a consistent manner. Evalua-
tion of the scalability of alternate communication im-
plementations within the middleware is ongoing. The
evolving tool kit will also enhance the programmability
of this simple, yet powerful distributed programming
environment.
References
[1] JavaSpaces Service Specification, Version 1.1, Sun Microsys-
tems, 2000, p. 20.
[2] Jini Network Technology, http://wwws.sun.com/software/
jini/whitepapers/jini-datasheet0601.pdf, 2001.
[3] CORBA 2.6.1 Specification, http://www.omg.org/technology/
documents/formal/corba 2.htm, 2002.
[4] D.D. Corkill, Blackboard systems, AI Expert 6 (9) (1991)
40–47.
[5] K. Decker, A. Garvey, M. Humphrey, et al., Parallel Blackboard
System, University of Massachusetts Technical Report 90-54,
1992.
[6] B. Hayes-Roth, BB1: An Architecture for Blackboard Sys-
tems that Control, Explain, and Learn About Their Own
Behavior, Stanford University Technical Report HPP-84-16,
1984.
 B.M. Stack et al. / Future Generation Computer Systems 22 (2006) 88–101
[7] B. Hayes-Roth, A blackboard architecture for control, Artif.
Intell. 26 (3) (1985) 251–321.
[8] B. Hayes-Roth, K. Pfleger, An Introduction to Blackboard-Style
Systems Organization, Stanford University Technical Report
KSL-98-03, 1998.
[9] S. Jenks, T. Domae, A blackboard approach to advanced avion-
ics systems, in: Proceedings of the 1994 Society of Naval En-
gineers Conference, Indianapolis, IN, 1994.
[10] S. Jenks, J-L. Gaudiot, Nomadic threads: a migrating multi-
threaded approach to remote memory accesses in multiproces-
sors, in: Proceedings of the 1996 Conference on Parallel Archi-
tectures and Compilation Techniques (PACT’96), Boston, MA,
1996.
[11] S.F. Jenks, H.L. Logan, Towards a more affordable avionics pro-
cessing architecture, in: Proceedings of the National Avionics
Conference, San Antonio, TX, 1993.
[12] F. Londono, A blackboard framework to support concurrent
engineering, Ph.D. Thesis, West Virginia University, 1990.
[13] D. Longstreet, Function Points Analysis Training Course, revi-
sion August 2002, http://www.softwaremetrics.com/.
[14] J.W. Mcmanus, Design and analysis techniques for concur-
rent blackboard systems, Ph.D. Thesis, College of William and
Mary, 1992.
[15] J.W. Mcmanus, J.A. Kaplan, W.L. Bynum, Automated Concur-
rent Blackboard System Generation in C++, Langley Research
Center NASA Technical Document NASA/TM-1999-209128,
1999.
[16] R.L. Mitchell, Search engines break the sound barrier, in: Com-
puterWorld Magazine, 2002.
[17] A. Newell, Some problems of the basic organization in problem-
solving programs, in: Proceedings of the Second Conference on
Self-Organizing Systems, 1962.
[18] H.P. Nii, Blackboard application systems and a knowledge en-
gineering perspective, AI Mag. (1986) 82–107.
[19] H.P. Nii, The blackboard model of problem solving and the
evolution of blackboard architectures, AI Mag. 7 (3) (1986)
38–53.
[20] O. Selfridge, Pandemonium: a paradigm for learning, in: Pro-
ceedings of the Symposium on the Mechanization of Thought
Processes, HMSO, London, 1959.
[21] C. Spruit, Blackboard systems in SOCCA: process evolution vi-
sualized by reproductive, communicating blackboard systems,
M.S. Thesis, Leiden University, 1997.
[22] B.M. Stack, S.F. Jenks, A middleware architecture to facilitate
distributed programming, in: Proceedings of the 2003 Interna-
101
tional Conference on Parallel and Distributed Processing Tech-
niques and Applications, Las Vegas, 2003.
Brian M. Stack received his BS degree in
Electrical and Computer Systems Engineer-
ing from Rensselaer Polytechnic Institute in
1998. He worked extensively as an operat-
ing system and security expert at Lockheed
Martin for 3 years. He was then awarded
a fellowship and worked as an RA at the
University of California at Irvine where he
completed his MS in computer network and
distributed system in June of 2003. He cur-
rently is a distributed systems architect as
Northrop Grumman. His interests include distributed systems, secu-
rity, and the future of open-source technology as a viable business
model.
Gene Hsiao is an undergraduate researcher
in electrical engineering and computer sci-
ence at University of California, Irvine. His
current research interests are in parallel and
distributed computing, and computational
science. He is a member of the ACM, Eta
Kappa Nu, the IEEE Computer Society, and
Tau Beta Pi.
Stephen F. Jenks received the BS degree
from Carnegie Mellon University in 1987
and the MS and PhD degrees from the Uni-
versity of Southern California in 1992 and
2000, respectively. He designed advanced
avionics and medical systems for 14 years at
Northrop Grumman Corporation. He is cur-
rently an assistant professor in the Depart-
ment of Electrical Engineering and Com-
puter Science at the University of Califor-
nia, Irvine. His research interests include
parallel and distributed computing. He is a member of ACM and
the IEEE Computer Society.