You are on page 1of 10

> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) < 1

System of Systems architectural considerations


for complex environments and evolving
requirements
Michael A. Corsello, Member, IEEE

human constraints imposed upon the overarching system in


Abstract—Complex “system of systems” architectures are terms of competing vendors, inter and intra-organizational
subject to a myriad of issues arising from the dynamic politics and security to name a few. It is extremely beneficial
interoperability these systems are intended to provide. Many of to develop a plan for designing the architecture so that it
these issues can be addressed or avoided by considering the
messaging interactions between system nodes prior to and during
addresses these anthropogenic constraints prior to the start of
the construction of the constituent systems. Standardization of the technical design phase of the project.
messages and interfaces is an ideal way to provide a consistent,
vendor agnostic vehicle for interaction and interoperability of The remainder of this paper will focus on both the technical
systems in this class of complex architectures. and human aspects of system of systems architectural
planning. The intent of this paper is not to propose an
Index Terms—standardization, system engineering, system architectural pattern or practice, but instead to illustrate a
integration, system of systems
conceptual plan of execution for the overarching system
planning and design that can reduce issues with development
I. INTRODUCTION by addressing each of these human factors in a technical
construct.
concept of a “system of systems” is quite old. A real-
T HE
world example of this concept is a city being viewed as a
system composed of independent but interacting systems. The
II. NON-TECHNICAL CONSIDERATIONS

electrical, water and sewer systems that serve a city are all A. Core Purpose
independent systems that provide services to the city. This is In the design and development of a system of systems, as in
the essence of a system of systems architecture. The concept the design of any system, it is imperative that there is a clear
of the collective whole (a city) being viewed as a system statement of purpose. Prior to initiating any substantive work
composed of independent systems (electrical, water, sewer) on the SoS project, it needs to be clear what the intent of the
can be nebulous in this construct, but the litmus test of a project, and ultimately the system, really is. No system,
system of systems is that the constituent systems contribute to regardless of size or complexity, can do all things. It is
the whole through some physical interaction. Another therefore of critical importance to have a scope for the project.
important point is that these physical interactions are provided It is this scope that will serve as the focus of analysis to
at discreet points (e.g. a fire hydrant or an electrical outlet) determine which pre-existing systems should be considered
which hides or abstracts the complexities of the independent for inclusion in the project as constituent systems.
systems that contribute to the larger whole. It is important at this time to qualify what a constituent
system is. Each system that participates in the larger system
A key aspect of many system of systems (SoS) of systems is considered a constituent system if it is directly
development projects is the nature of the environment in part of the overarching system of systems. Any system that
which the individual systems are constructed. It is common supports a constituent system, but does not directly support
that one or more of the constituent systems are pre-existing the overarching system of systems is a subordinate system.
(legacy) and complex in their own right. This means that the Regardless of whether a system is a constituent or subordinate
concept of connecting these individual systems to a larger system, it can still be critical to the successful execution of
collection of systems is both an integration challenge and a one or more capabilities provided by the system of systems.
potential source of discontinuity to the purpose of the In a system of systems architecture, constituent systems can
overarching system. serve various roles:
Beyond the technical aspects of the architectures there are --Core system, which is required for the functioning of
the overarching system. This is an infrastructure provider
such as a directory service or security provider.
Manuscript received September 17, 2007.
Michael Corsello is with Booz Allen Hamilton, Lexington Park, MD --Member system, this is a “first-class” system in the
20653 USA 703-618-3087; (e-mail: michael.corsello@ieee.org).
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) < 2

overall architecture which fully participates in the operation of in a consistent manner that constituent systems can be adapted
the overarching system. to (via individual integration projects). If these interfaces are
--External system, is a system that does not participate designed based upon any specific vendor’s pre-existing
fully in the overarching system. This class of system can be system, there is likely to be vendor and organizational
considered a “consumer” that receives benefit from the disharmony due to these biases. It is therefore important to
overarching system but generally does not provide to the either adopt existing industry standards from known standards
system. organizations (e.g. IEEE, ISO, W3C) or to develop project
Of important note here is that any system may indirectly specific standards via a collaborative effort with the
provide to the overarching system, but if that system provides participating vendors and sponsor organizations.
or collaborates through another system, it may not be Of specific note, the creation of these interface standards
considered as part of the overarching system (in which case it should include data standards such as encoding and logical
is opaque to the overarching architecture). data model standards as well. By maintaining vendor
neutrality to this extent, each participating system in the
B. Organizational Support
overarching system will have to develop an integration
The success of any project will depend on the support of strategy to only the interfaces and logical models which can
participating organizations. In system of systems projects, happen in isolation from the overarching group.
there are generally multiple organizations that participate,
many of which may be direct competitors. In this type of D. Organizational Politics
project it is not uncommon to have multiple stakeholder The development of any system will be affected by support
organizations that have a vested interest in the system being from the participating organizations. Even in small projects
implemented. These organizations may be competing for that support only a single organization, departments within
additional resources or influence on the direction and scope of that organization may disagree or compete for resources or
the project. Each of these organizations will also have representation in the design and development of the system.
existing systems that support the current means of conducting In complex system of systems architectures this is generally
business that are to be addressed in part or in whole by the amplified many times over. It is not uncommon for a system
new system of systems. This therefore means that these of systems to support not only multiple organizations, but
organizations have an interest in their current systems being many of those organizations may be multi-national or even be
adopted as part of the architecture. multiple national governments. This raises issues of
For each of the current systems used by participating organizational or even national politics over the interactions
organizations, there may be a different vendor that produced and sharing of capabilities between these organizations. There
that system, resulting in direct competition for vendor support are concerns over financial responsibilities, legal or regulatory
of constituent systems. Each of these systems must be compliance (and compatibility) and of course security.
considered at least as candidates for inclusion in the In the design and development of a system of systems that
overarching architecture if they fulfill some portion of the is constrained by this type of environment, the isolation of
requirements covered in the scope of the project. individual constituent systems may need to be considered to
By maintaining a consistent focus on the scope of the support the diverse set of rules that these political constraints
project for the overarching system, it is easier to form may require. In these scenarios, an additional layer of
quantitative criteria to determine which of these pre-existing complexity may need to be added to address these constraints.
systems should participate in the project and in which role. To address this form of political and legal complexity, the
These determinations tend to be political in nature because design phase of a system may best be served by design
each participating organization has a vested interest in the governance communities that can evaluate the planned
adoption of their current system(s) as a constituent system. architectures to ensure that each organization’s political and
This often results in the “marketing” of a system that may only legal needs are adequately represented.
partially fulfill any requirement to be addressed by the new When a system engineering project of any type involves
system of systems. The quantitative evaluation of systems multiple organizations, financial responsibility can become an
nominated as candidates for inclusion in the architecture is issue. In terms of a project that spans organizations of varying
therefore key to maintaining support from member financial means, such as in a multinational project that is
organizations. intended to support nations of high and extremely low gross
national means, special consideration must be given to reduce
C. Vendor Neutrality
the cost of participation. In some cases this may include the
In the construction of a system of systems, each system role incorporation of long-running transactional capabilities that
can be categorized by a set of intended interactions. This set permit human transfer of data via means such as CD, DVD,
of interactions is the basis for the overarching system interface removable drive or tape. The implementation of these means
or application programming interface (API). in a system of systems construct should be up to each
Given that there are likely to be many competing vendors, independent constituent system, but the interaction of these
each of which may have products that are fundamentally systems may be affected at the overarching level.
incompatible, it is imperative that these interfaces are defined To accommodate these types of barriers to entry for
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) < 3

participating organizations, each capability to be provided by security is the realm of enforcement and the definition of who
the overarching system should have a concept of operations can enforce and where enforcement can be conducted. It is
that addresses these organizational limitations. It may be common that security will be isolated within each constituent
directly addressed via financial support of shared technology system and a set of security capabilities will be additionally
infrastructure being provided by a weighted “means to pay” applied in the overarching system. The design considerations
participation schedule. These forms of financial and politics of which are too complex to cover in this paper.
considerations may be crucial to the success and longevity of
the project and must be considered from the inception of the III. TECHNICAL CONSIDERATIONS
project. System of systems architectures are generally very complex
In this form of project it may be desirable for the more and bring together a collection of new and existing systems
financially able organizations to take on the entire cost of the within a loose framework of overarching technologic
project, or to only share costs across the most able capabilities to fulfill a larger set of requirements.
organizations. While this seems to be a socially responsible As in all system engineering projects considerations have to
action, it may well alienate the less wealthy organizations as be made in regard not only to the capabilities required, but
being second-class participants relegated to only having access also to the capacity required for each capability. This may be
to what is provided by the paying parties rather than actively realized through the need to host multiple instances of a
contributing to the process. specific system type or (capability class system). In the case
E. Security of any distributed form of system, the overall capability of a
system of systems is hosted on multiple nodes, each of which
In the development of multi-organizational systems security
will host one or more constituent systems (or capabilities).
is a critical factor to be considered. Each participating
organization may have limited trust of the other organizations. When transforming to a system of systems an increase in
demand for each constituent system should be anticipated as a
This may not be a limited trust of the organization as a whole,
result of the larger community, this may result in a need to
but instead be a limited trust as it pertains to the member
increase the number of nodes for each constituent system.
personnel, data or business processes and methodologies
codified in the systems being constructed. In this way, each
A. Constituent Systems
organization’s security requirements will have to be addressed
individually in their own systems, in the constituent systems In the construction of a system of systems, each of the
of the architecture and in the overarching system of systems individual systems that participate constitutes some portion of
what is required to fulfill the overall requirements. It is not
core architecture.
uncommon also that there are multiple individual constituent
Aspects of Security systems that all provide the same capability, potentially from
System architecture is subject to multiple aspects of security different vendors with slightly different algorithmic
including: implementations. When interfacing these constituent systems
--Authentication, is this actually “who” they claim to be into the system of systems construct, it is important that they
--Authorization, is this “entity” allowed to: all connect via a common set of interfaces provided through
--Access, a specific resource, data or otherwise the overarching system.
--Perform, a specific action, such as executing code
--Non-repudiation, this “entity” requested this access or Legacy Systems
activity at this time with these consequences Each legacy system that participates in a system of systems
These aspects are the basis for security considerations and may in fact contribute a unique capability only fulfilled by that
may apply to multiple sections of the overarching system as specific system. To accommodate the increased demand for
well as to the constituent systems in the architecture. In a that capability, which results from expanding the user
system of systems construct this can have significant community to the full system of systems user base, additional
ramifications as to the control and partitioning of authority for nodes may be required. In the determination to include such a
the designation and maintenance of these permission sets. legacy system in the overarching architecture, it is important
to consider the strategy by which such nodes are added.
Trust Questions to consider in this scenario include:
Expectations of trust between organizations within the --Internodal activity, was this legacy system constructed
context of a system (or a system of systems) is no more to support additional nodes for the subordinate systems this
realistic than in any other context, so trust will tend to become system relies upon?
a key factor in the design of security features and in the --Platform support, is this legacy system supported on
segmenting of system capabilities in the interactions of equipment that is still available?
constituent systems. --Source access, is the original source of this system
available for consultation to address scalability needs?
Enforcement and Authority
An important consideration in the implementation of The legacy systems used in a system of systems architecture
will often provide the backbone of the initial operating
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) < 4

capability (IOC) when the overarching system goes live. It --Data services, it is not uncommon that “cloud based”
should therefore be anticipated that it will be necessary to data services will be required to facilitate the dissemination of
expand these legacy systems to accommodate additional information to other services. This may actually be a pass-
capabilities and capacities as needed. These expansions through service to other member systems.
should be planned in advance to ensure operational --Generic brokers, are services that provide a central
capabilities and capacities are always available when needed. location to request some specific service that is then arbitrated
between actual providers of the service. An example of this
New Systems capability may be a round-robin load balancer.
In the case of the development of new systems to provide --Informational services, are generic services that provide
some required capability, there are still the traditional a catalog of informational constructs such as service schemas,
constraints imposed by the hosting organization’s system bindings, etc. These services are quite diverse and
infrastructures. These constraints will need to be addressed may include custom capabilities unique to the domain of the
during the development of those systems and may result in overarching system.
limitations to the deployment of multiple instances of this
C. Member Systems
system to different hosting organizations. This underlying
system infrastructure constraint may result in increased cost, All constituent systems that are not providing core
time and development efforts to support multiple platform capabilities are considered member systems. This is the
implementations of a capability to achieve the desired majority of systems in the overarching system. Any
capacity. functional capability that is offered by the overarching system
One of the aspects that favor the development of new is provided via a member system. These member systems
constituent systems to support a system of systems is the lack may themselves be distributed to provide an enclave of
of imposed constraints on technical implementations other capability that is exposed to the overarching system as a single
than environments. This enables new systems to follow the entity (as in load balancing), or distributed as distinct nodes of
current industry best practices and the specific guidance of the capability that are integrated by the overarching system. Each
project architecture without the need for a specific integration of these patterns of distribution for a capability has different
layer to connect the system to the overarching architecture. impacts on the larger system design, including monitoring of
Even in these systems however, it is a best practice to still health and capacity and more significantly, data
abstract the interfaces and interactions with the overarching synchronization across nodes.
system to enable external changes to the requirements to be If the nodes that provide a capability are exposed
implemented as close as possible to where they occur. separately, it is possible (as noted earlier) that each node may
have a specific implementation that will yield a slightly
different result when used. In this circumstance, the adhesion
B. Core Capabilities of a client request to a specific node implementation may be of
The system of systems is defined as such by the fact that significant importance for valuation integrity. This is
each of the constituent systems, regardless of role, has some accomplished through algorithmic implementations of the core
interaction with one or more of the other constituent systems services, but needs to be accounted for early in the
in the overarching system. This basic requirement indicates architectural design phase.
that there is generally some amount of arbitration or Each hardware node may provide multiple capabilities to
orchestration at a minimum that must be provided. All of the overarching system, potentially including core capabilities
these infrastructure requirements are provided as core as well as functional capabilities. In this case, it is important
capabilities through core constituent systems. to know the hardware bindings of capabilities for operational
In architecting a system of systems, each of the assurance as covered in the “External Systems” and “System
requirements for infrastructure support will be bound to one or of Systems Infrastructure” sections of this paper. If a single
more of these core systems. It is generally best to architect hardware node is implemented in a load balancing scenario,
core capabilities in the most flexible, adaptable and distributed multiple physical nodes can act as a single node to provide
manner to ensure the greatest level of scalability for these greater capacity and reliability for the services hosted.
systems. This often means that these core capabilities are However, accounting for this configuration is difficult for
mapped on a one-for-one basis to a system. It is also generally management interfaces and may affect the overarching system
best to leverage existing standards for these systems especially operation if not adequately accounted for (e.g. sessions for
to maximize the likelihood for current support from existing long-running transactions managed externally of the node).
constituent systems while minimizing the likelihood of vendor When constructing member systems consideration must be
biases. given to all aspects of the potential environments these nodes
Common core requirements will include capabilities like: will be hosted in. This includes all aspects of hardware
--Login/Authentication services configurations, network defenses, systems management at the
--Directory services, this is a broad category that includes node level, availability (especially in poor regions of the
organization directories, vendor capability directories, system world), legacy software bindings, software implementations,
directories as well as component or services directories. etc. The construction of a distributed system is always
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) < 5

difficult and the addition of the dynamic environments often logical data types to a domain specific data type for each
extends these difficulties for system of systems architectures. domain.
--Logical associations, including rules on how logical
IV. STANDARDIZATION data types can be composed to create business data entities
In the development of a system of systems it is critical that --Rules, including how data can be converted between
all inter-system communication is standardized to facilitate logical representations and how to represent a vendor specific
integration of the constituent systems. Many times, the binding.
overarching system of systems will be based on standards that --Encodings, how to physically represent information in a
conflict with the standards already implemented by some of file, stream or opaque entity.
the constituent systems. Or put another way, it is likely that B. Standardization Process
the constituent systems will be based on established standards In order to exchange information (data) with another person
that are different but overlapping. The overarching system or organization, it is imperative that both agree upon the
will have to adopt a single standard for each interaction or set information to be exchanged. This agreement is the essence of
of interactions that all constituent systems will have to comply data standardization and applies equally well to distributed
with. This is both a technical and non-technical issue, as the components, systems and system of systems.
existing standards may be fundamentally incompatible, but There are many forms and levels of data standardization,
must be adapted to interoperate. each of which has certain benefits and limitations to the
The standardization of a system’s interactions should be parties involved with the data being standardized. At each
based upon established, open, supported standards. It is often level of standardization an increasing level of understanding
difficult to find standards that can be adapted to suit the and interoperability can be obtained. This standardization is
overall needs of the larger system of systems and the existing much the same as in spoken languages, where each area of
domain specific standards already in use by the constituent subject matter has its own vocabulary. As is also true with
systems. This is a problem domain that is not directly spoken languages, each isolated group of speakers will
solvable, but instead is worked around through careful and develop a vocabulary that is distinct but similar to the other
thorough design. In this area, the standardization of message groups.
interchanges and data models is the key to overall success. Without standardization of data, as with languages,
A. Standardization Areas communities that do not “speak the same language” cannot
Some of the fundamental reasons for standardization in any effectively communicate. These linguistic traits exist across
system include: dialects of languages as they do across user communities,
--Upgrade paths, enable the upgrade of systems and there may be enough commonality to “get by” in primitive
standards between versions in a dependable, repeatable understanding but at a tremendously increased cost. When
fashion dealing with system of systems architectures, this
--Extensibility, of the systems and standards to understanding comes in the form of semantic understanding of
accommodate changing requirements data between the member systems and components within
--Customization, of the implementations participating in these systems. These issues of semantic interoperability must
the system (Black-Box) be addressed to allow disparate (especially legacy) systems to
--Translation, by external organizations and vendors of communicate in order for the overarching system to function
both data and programmatic messages effectively.
--Integration, of third-party systems, components and C. Communities of Practice
data now and in the future A community of practice (CoP) is the collective grouping of
To accomplish these goals, the standardization process must all people and organizations that share a common goal or have
be well thought out to avoid these common pitfalls: a common need. In terms of data standardization, these CoP
--Complexity, of data types and procedures that result members deal with the “things” that map to, or are described
from standards developed at a high-level. by, data.
--Evolution, of requirements and technology over time Data standardization is often performed by working groups
which will likewise require additions and extensions to (WG) within a CoP for data specific to that CoP. Many
standards. distinct CoP may deal with similar or overlapping “things”,
--Perspectives, of the communities leveraging the each of which need to be described separately. In a system of
standards; concepts need to be translatable. systems architecture the development of data working groups
In order to achieve success in a system of systems and general CoPs are critical to the successful integration of
architecture the standards used need to address all of these the member systems. This often means competing contractors
areas. A list of domains that may be standardized in this must come together to develop a common set of data standards
manner includes the standardization of: all groups can operate with for the overarching system.
--Logical data models, including primitive data types, With an understanding of the scope of data to be
entity types (e.g. tables) and complex structures (e.g. graphs, standardized (data relevant to the CoP), the process of
trees). defining the appropriate standard is possible. At this point, it
--Business domain perspectives, or profiles that map is advantageous to examine several points that will serve to
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) < 6

define the standardization framework and ensure that there are separate terms. In the latter case (first: John, last: Doe), the
common goals for the standard. The goals of a data standard full name can be trivially derived by concatenating the 2
are most commonly: entities in the order given. In the former case, this is not as
• Enable data sharing between members of a CoP trivial, even if the description of full name defines that the
• Enable data sharing between related CoPs (refined in order of the internal sub-terms are first then last. It is
later sections) plausible to add another wrinkle to the name concept, by
• Enable consistent storage of information adding a middle name that is potentially non-existent. Further,
• Enable consistent, quantitative, comparable results from it is reasonable to expect that the use of a middle initial alone
all members of the CoP is allowed instead of the full middle name. If the first, full
• Achieve maximal value from data stored middle and last names are provided by one organization, any
• Achieve minimal redundancy of data generation within other of the combinations of name can be constructed, thereby
the CoP meeting all other organizations needs with only a trivial cost at
• Achieve maximal data consumption from other CoPs deriving the additional forms of name. In the case of an
Each of these goals has a cost associated for generating a organization that requires the first, full middle and last name
standard that meets this goal, and for generating a standard elements, it is non-trivial for that organization to consume the
that does not meets this goal. It is important to realize that information from other organizations that produce the more
once a standard is finalized, it should not be changed until generic full name entity alone.
there is a truly compelling reason to do so. Even then, the Communities involved in a subject area will tend to have
standard should be changed only to encompass the new multiple information producers, especially prior to
community requirements while maintaining as much of a standardization. These information producers will need to
compatibility with the existing standard as possible. This negotiate the level of responsibility each should assume to
exemplifies one additional cost of standardization; the ensure that a single core source of truth exists for any single
potential reduction of flexibility. instance of a subject item. An example of this for records on
In order to accommodate emerging requirements of the people might be a single organization that has the
CoP, a standard needs to be as forward thinking as possible responsibility of defining the creation of a person’s record.
since the standard will always take time to be developed and That record will be the authoritative source defining the
adopted. If the CoP that develops the standard strives to stay existence and record of that person. This source may be
ahead of the curve and aggregates requirements to the greatest defined in the community as the hospital at which a person is
extent possible, the lag caused by standardization can be born or first recorded (i.e. in the case of a home birth). The
minimized resulting in negligible effects on the practicing manner in which this record is created and stored for
community. information exchange is relatively insignificant as long as the
When developing standards for a system of systems it is record creation is performed in accordance with the standard
possible that some of the goals of standardization apply at for the data being created. The fact that any of the thousands
different levels within the overall architecture. If multiple of individual hospitals may be the site of a specific birth is
nodes need to store the same data, but do not interoperate at inconsequential as all of the hospitals are using the same
the level of the stored data (instead exchanging data at a standard for data collection and recordation. Each hospital
service interface), the goal of consistent storage may be local may have its own unique manner or process for the collection
in scope to each participant rather than global to the and recordation of the information, but the final disposition of
overarching system. that information is based upon the standard, ensuring that the
D. Granularity of Discourse information is available and shared across the community.
If a subject matter area has a well defined set of terms with Once a level of standardization for data entities used in a
definitions, discussions (information exchanges) can occur system of systems are agreed upon, the integration of those
with meaning being implicit given that the parties all share the constituent systems can occur. This integration adds its own
common set of terms and definitions. This does not however, complexities in addition to the complexity of standardization.
mean that all parties gain the level of meaning that they
require to accomplish their intended operations from the
information provided. The extraction of value (meaning / V. INTEGRATION
understanding) from information is additionally constrained The foundation of any distributed system is based upon the
by the detail or granularity of the terms used in the integration of information in some form across system
conversation. boundaries. This integration can be accomplished simply
In most communities, a set of terms will generally be through procedure calls, or specifically through the
provided that are telescopic or hierarchical in meaning. These marshalling of information used as inputs and outputs of
terms can be exemplified by the concept of names for people. remote procedure calls. Or, in a more complex environment
In human names, there is the basic concept of a first and last this may include the integration of full data sets in some form
name. If a person is described in a conversation by their full between systems. A relational database management system
name (i.e. John Doe), as a single entity, that provides a (RDBMS) is a simple example of this complex form of
different level of granularity than if the name is provided as 2 interaction, where data is stored in the RDBMS and marshaled
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) < 7

into and out of the RDBMS to externally distributed systems --Interleaving, data from different systems are combined
that use the stored data. The implementation of most RDBMS into an integrated view of information, with corresponding
applications places the responsibility of translating (or data entities from each system matched to provide a view of
encoding) the data into a form the RDBMS understands on the the authoritative source of each record. Updates are
client system. This is a vendor-specific binding for that allowed locally but may or may not be allowed back to the
RDBMS application. In many system of systems architectures source. In this scenario, updates are generally local in
this type of application of technology would not be permitted nature and the updates are persisted as a delta from the
due to vendor-specific encodings on the “wire”. Instead, in source. This avoids the complexity of data interactions
this form of architecture the data could be translated into an required for dynamic updating.
intermediary form that is standardized, such as XML for
transmission between nodes. This is the approach taken by
many service-oriented architectures (SOA) including some
that are also system of systems architectures.
A. Data
Data integration is the most complex challenge to
integrators. Data can be voluminous in size which poses
transmission constraints. Data is structured differently for
different purposes which pose compatibility issues and it is
often complex in structure which causes access issues. The Figure 3. Interleaving
ultimate goal in a system is to process data to produce
information which a user can comprehend as knowledge. This --Dynamic, just as in the interleaving form, data is
makes the integration of data an important part of systems integrated from multiple sources; however, updates are not
interactions. All system interactions will involve some only local but transmitted to the source data system.
amount of data transfer, which will often involve a need for Additionally, inconsistencies between data sources for
data integration. There are multiple types of data integration matching elements may be reported.
that can be used in complex systems such as:
--Association, which has 2 basic forms:
--Overlay, data from different systems is not combined in
any fashion. Instead the data is displayed side-by-side from
the different systems. Updates may or may not be allowed.
This avoids all complexity of data interactions.

Figure 4. Dynamic

--Synchronization, all data from all systems are treated as


Figure 1. Overlay a single source with no distinction as to the originating
system. All updates are transmitted in real-time to all
--In-lining, data from different systems are combined into partners. This is a complex process that addresses the
an integrated view of information, but each source system’s issues of data interactions.
data is treated as distinct. Updates may or may not be
allowed. This avoids all complexity of data interactions
other than compatibility of data models.

Figure 5. Synchronization

In full synchronization, all updates are critical in time. The


operation of changing data can have cascading effects
Figure 2. In-lining
especially if multiple synchronization partners are involved
--Integration, which has 3 basic forms: with different areas of responsibility for subsets of data.
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) < 8

B. System Interfaces to be supported may have user interface considerations, data


Other than the data itself that is exchanged between latency or transaction time constraints and generally impose a
systems, the interfaces for calling procedures and requesting different set of requirements on the overarching system. This
informational resources is the primary means of interaction is often handled by specialized clients that serve as service
between constituent systems. The integration of systems brokers for the clients at large. This added level of indirection
occurs at the exposed system interfaces as this is all that is can impose additional legacy impacts that need to be
available to external systems. The interfaces may be exposed considered in the design phases to ensure these latencies are
through one or more protocol, such as the WS-* stack for web accounted for.
services using SOAP with bindings to XML over System interface brokers that support client interactions to a
HTTP/HTTPS. The bindings may be either generically textual system of systems will have to additionally serve as the focal
like in the XML over HTTP mechanisms or explicitly binary point for authentications and many other security aspects of
as in the use of CORBA, Java RMI, .NET remoting, etc. The the system. This can result in a need to scale these broker
design and development of these system interfaces should take systems more rapidly than expected in the design phase due to
into consideration multiple aspects of the interfaces user loads. The cost of such brokers will have to be
themselves, including the bindings we just covered. considered in terms of which organizations host the broker
Interface Considerations systems for a given client community, thereby incurring the
When planning for system of systems integration at the costs. Considerations also need to be given to concepts of
interface level, it is a good plan to anticipate a requirement to caching and integration of results at these client broker sites.
support multiple interface bindings for each interface Finally, decisions will be made in the design phase as to how
definition. In this way, a single system can have an internal clients can access the overarching system. The broker systems
implementation with multiple exposed bindings, or an may be the solely permitted client access points to the system.
orchestration system can provide proxy bindings to enable Or, no brokers may be used at all, in favor of a discovery
compatibility between incompatible systems. This is an service that is published at a specific resource node that
example of the concept of least capable design, where indicates where systems are accessible. The latter may affect
interfaces are designed in such a way as to maximize the the mobility of nodes if not further abstracted across the
bindings that can support the interface. This is further system domains.
supported via encoding standards, rule standards and business C. Dependent (Opaque) Systems
perspective standards.
Systems outside of the system of systems boundary that are
required for the successful operation of one or more nodes in
VI. EXTERNAL SYSTEMS
the system of systems (e.g. a database server, LDAP server)
In system of systems architectures there will often be are still considered external systems that the overarching
external systems that simply consume the services of the system is dependent upon. Dependent systems are often
overarching system. These external systems are considered overlooked in analyzing the supportability and dependability
clients of the system and bring their own concerns. of the overarching system. In many cases this is due to the
A. End users core design group(s) having lack of access to the individual
organizational units that are supporting the physical systems
Human users of a system of systems may or may not ever
that in turn support the overarching system. Since these
directly interface with any node in the system of systems.
systems are opaque in terms of access from the system of
Instead, human computer interactions (HCI) may be
systems as well as to the organizations involved in the design
accomplished via an externally connected system such as a
and development of the system, these systems are of special
web server via a web application. With the development of
concern to designers.
smart client technologies, which blend lightweight thick client
While full accountability in an architectural design is
applications with back-end systems providing the functional
desirable, the organizational politics and general logistics of
services, it is entirely possible that many HCIs will be directly
acquiring such information are often unrealistic. Instead, to
conducted with a service node via a smart client.
ensure that critical supporting systems are maintained and
In any manner of interaction, human end users need to
available as needed, service level agreements are used to
perceive that they are being provided value from the system
ensure whatever is not seen by the system design will still not
and that the system is responsive to their actions. This pair of
affect the availability of the overarching system. This form of
requirements may tend to blend the system of systems
assurance is still far from complete as it is enforced via
architectural designs with more traditional user level
levying fines or through other contractual means which may
application designs at these boundary interfaces.
still result in extended downtimes for the system users.
B. Clients
D. Availability
Clients of a system of systems may not be required to
The primary means of ensuring availability of system
conform to the same set of interfaces as the constituent
capabilities is via redundant nodes that offer redundant
member systems. In this manner, the public facing client
capacity of some measure beyond peak demand. This concept
interfaces are often less rich and more diverse. Clients that are
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) < 9

of course comes with additional cost for maintaining this oversight to visualize the availability and health of the
continuity of operations (COOP) capability, but will tend to overarching system. The area of distributed system
alleviate the problems that may arise when a required management is still being actively developed by many
capability becomes unavailable. In addition to providing a communities and progress is being made.
redundant capability, this capability should be adequately One means of managing this type of complex system is
separated by geography to ensure natural disasters would most through the use of agent services that communicate the
likely not disrupt all nodes providing the specified capability. availability of node networks and continually analyze that
When designing for COOP it is important to consider the information. This is accomplished via both statistical and
impact each decision made. It is not uncommon to implement genetic algorithms hosted on various core nodes in a system.
COOP as a manual process such that when a node fails, users Projects like Globus, JXTA, BOINC and groups such as the
are expected to map to the redundant node manually with an Global Grid Forum are advancing these technologies for
expected latency in data interactions that may result in a loss inclusion in complex distributed architectures, including ones
of up to several days’ worth of data if a failure occurs. In that are based upon a system of systems approach.
many system of systems architectures, no system failure is A. Considerations
allowed to impact the operation of the overarching system, In order to manage a system of systems it is important to
including no tolerance for any data losses. This greatly affects know the status and health of each capability and the capacity
the design of the constituent systems to ensure that all nodes utilization of that capability more so than to know the status of
can interact and exchange information in near real-time so that the nodes providing that capability. This is a paradigm shift
data is committed to multiple nodes prior to being considered from traditional system management where it is the machines
stored. This is a large design consideration that must be made that are managed and not the capabilities they provide.
prior to construction of the system nodes and must be When a system is composed of nodes that each provide
considered for supporting legacy systems as member nodes in some unknown number of capabilities (perhaps beyond that
the architecture. available to the overarching system), it is difficult at best to
Mechanisms such as coherent caches, canonical sources or measure the health or availability of the capability per
chains of custody may provide a timely solution to the data machine or in total. Instead, a heuristic method is required
distribution and integration challenges arising from dispersed that essentially learns the current capability and capacity with
storage nodes. This is a fundamental area of research that respect to the variations of that availability. This form of
poses tremendous challenges to distributed systems in general. analysis can then be deployed as a capability on each node that
COOP is a complex process in itself regardless of the will accept such analytical components. This analytical
systems being “COOPed”. When designing a system of component can then isolate the variations of availability to
systems, this complexity is amplified by the factor of the arrive at a continual metric of health. This form of distributed
number of constituent systems involved. In a system of analysis should yield better results with larger networks and as
systems COOP strategy, it is important that a single system such be a standard distributed mechanism for systems
failure or overall single site failure not disrupt the operation of monitoring.
the overarching system. To accomplish this it is critical that The management of the nodes providing a capability will in
an “autonegotation” is performed by all nodes with regard to general not be solved via technologies (that is already done
all capabilities the node is required to consume. This well) instead, node management will be a responsibility
“autonegotiation” can be performed in any number of ways assigned through policy. In simple terms, the owner of each
including though a DNS round-robin, load balancer, physical hardware resource will determine who will be
messaging bus or other underlying system provided permitted to manage that resource. Many organizations are
capabilities. It is important however to consider the impact of not willing to permit management level access to personnel
failure of these COOP enabling components as well as the not affiliated with their organization.
functional systems that perform the work. Finally, the
development of COOP may require further customizations to VIII. ADDITIONAL CONSIDERATIONS
system components that alter system design specifically to Due to the multi-organizational participation in system of
support COOP in certain environments (such as SAN systems projects, there are additional considerations that must
replication of nodes). be addressed.
A. Non-Technical
VII. SYSTEM OF SYSTEMS INFRASTRUCTURE The organizations that participate in a system of systems
The management of complex systems that are loose project will generally each have their own vendors. It is also
aggregations of other independent systems is a daunting task. common that vendors may be involved in this class of project
Moreover, there is often no one organization overseeing the directly as well. In this case there are competition issues that
system at large as it is not physically a resident system need to be addressed. Generally, the vendors may be reluctant
anywhere. Instead, management of the infrastructure of these to adopt and reuse systems and system components that were
systems is generally accomplished on a nodal basis, where developed by another vendor. This may be due to a perceived
each node is managed individually by the custodian of a loss of revenue from the development time, or due to a lack of
specific node. This is cost effective but lacks a central trust toward the other vendor(s). In either case, these types of
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) < 10

issues need to be addressed early on in the project. One participating platforms may need a custom set of bindings or
simple way to address these concerns is via group adoption, APIs to enable this capability. If this is done, then those
where no single organization can make any decision without specific bindings should be standardized in addition to the
approval of an oversight group. This oversight “red tape” regular APIs in use.
comes with its own costs. Documentation is a key factor for technical success and
Vendor biases can also be seen in other organizations longevity of a complex system. Given that most system of
through the “not-built-here” or “not-bought-here” syndromes. systems projects will take many years to complete, the change
In these cases, an organization simply will not allow their in personnel and organizations involved necessitate the
developments to rely on components they did not build or generation and maintenance of good documentation. This
purchase. This is a difficult concept to overcome, as this is an documentation must be produced throughout the course of
emotional issue rather than a technical one. It may be possible design and development. Once produced, all documentation
to address this type of consideration through group approvals must be distributed such that it is available and discoverable
as well. for those who need it.
Deadlines are a fact of life in all projects. In system of
systems engineering projects, these deadlines can make or IX. CONCLUSION
break the project as a whole. Some simple guidelines that can A system of systems is a form of distributed architecture
help a project to be successful include: that commonly leverages existing systems to provide
--Small scope, for each item planned. If a deadline is functional capabilities that the overarching system of systems
missed, it is on a small unit of work with lesser impacts. will then expose externally. This form of architecture is often
--Small teams, for each item planned. If a team is small that intangible and will tend to be more constrained by political
works on an item there is less likelihood of disagreement in and organizational considerations than other forms of
the team. distributed systems. The development of these systems
--Under sell, never promise more in a single work unit than requires a full complement of systems engineers, business
what is required. managers and project management black belts to be truly
--Plan ahead, the more future units of work that are planned, successful.
the less problems should arise. As with any systems engineering project, system of systems
As in any multi-organizational project, distribution of labor engineering requires comprehensive planning to form an
is a consideration. When varying levels of financial adequate design that can be realized as an effective system.
availability are also a factor, the distribution of labor can be Additionally, a system of systems can be of even more benefit
additionally impacted. It is therefore beneficial to plan labor than smaller systems through the use of parceled and well-
based upon unit of work assignments and available funding. planned standards.
When financially constrained organizations are the ones with a Overall system of systems architectures are about providing
primary responsibility in a functional domain, concessions extended value through the leveraging of multiple
will have to be made to ensure these organizations can fulfill organizations’ existing resources, often including their legacy
their responsibilities. This type of issue is a possibility, a plan systems to provide keystone capabilities.
should be devised in advance to handle such problems if they
arise. This may be an external pool of funding such as REFERENCES
“membership dues” for organizations or designations of “pay [1] Bar-Yam, Y. and others. “The Characteristics and Emerging Behaviors
for service” where all prospective clients of a capability are of System-of-Systems” in: NECSI: Complex Physical, Biological and
required to pay a portion of the development costs. Social Systems Project, January 7, 2004.
B. Technical [2] DeLaurentis, D. A. and Callaway, R. K. “A System-of Systems
Perspective for Future Public Policy,” Review of Policy Research, Vol.
In any system of systems project, there will tend to be 21, No. 6, 2004. pp. 829-837.
multiple platforms being used. This includes both hardware [3] Kotov, V. "Systems-of-Systems as Communicating Structures," Hewlett
and software platforms for nodes and networks. As such, Packard Computer Systems Laboratory Paper HPL-97-124, (1997), pp.
1–15.
there are considerations that need to be made throughout the [4] Wikipedia contributors. System of systems engineering. Wikipedia, The
design of the overarching system to ensure that any reasonable Free Encyclopedia. April 30, 2007, 04:44 UTC. Available at:
set of technologies can be supported in the system. While http://en.wikipedia.org/w/index.php?title=System_of_systems_engineeri
ng&oldid=127048610. Accessed September 16, 2007.
there are simple decisions that can be made to support this [5] Wikipedia contributors, System of systems. Wikipedia, The Free
goal universally, these decisions also have to take into Encyclopedia. Sep. 16, 2007, 15:43 GMT. Available at:
consideration the performance and cost impacts. http://en.wikipedia.org/wiki/System_of_systems.
Cross-platform and cross-language interoperability is a
major consideration for system of systems architectures. If
standardized interfaces are well designed, this may be
alleviated in the overarching system, but may still be an issue
within the constituent systems. Additionally, for performance
reasons, some internode communications may be designed to
operate “outside” of the standard interfaces. In these cases, all