A Playbook for ESBs

Co-authored by Progress Software

1 ebizQ copyright 2007

Executive Overview

Over the past few years, enterprise service buses (ESBs) have emerged as key enablers for more agile
and effective integration across enterprise systems and business processes.

Yet, as a technology, ESBs are only part of the answer. What is required to deliver agile and effective
integration is a combination of the right ESB technology with “defined patterns” that increase
effectiveness of the integration team in their use of an ESB. Developing and applying integration
patterns can result in an exponential improvement over once-in-a-row integration approaches. In
particular, there are four common integration patterns with an ESB. By recognizing when these patterns
apply to their projects, integration teams can deliver projects and business value in a highly reproducible
(and therefore risk-free) fashion.

This paper introduces four integration patterns for ESBs and the enterprise integration scenarios they
help make possible. Much like templates for generating code modules, integration patterns enable
organizations to reuse code and configuration elements to maximize the deployment of integration
projects quickly. Combining the advanced capabilities of an ESB with integration patterns can deliver a
number of key benefits to both business and IT: reduced risk, faster time to business value, greater
agility and lower costs.

Organizations considering (or already using) ESBs should consider the types of integration patterns
most suitable for their ESB. By doing so, IT leaders gain an understanding of the potential impact a
given ESB can have on their enterprise integration backlog, and ultimately, the rate of delivery of new
business functionality stemming from that ESB.

Making Integration Practical and Effective

In the world of enterprise computing, the life of the IT manager keeps getting harder. CIOs and IT
leaders are responsible for designing new technology solutions that operate efficiently and work with the
existing IT infrastructure, yet they also need to support the business changes that the line-of-business
managers demand. While the recent focus on shrinking corporate budgets may lead you to believe that
it’s the CIO’s job to focus on the efficiency of an IT organization, it’s not. That’s merely one requirement.

Instead, CIOs and IT leaders today should be focusing on enabling the business to become a better
business, through the application and effective use of technology. In essence, CIOs have become
technology brokers to the business world. Instead of being tied up in the technology itself, IT leaders
have become the translators of business strategy into technology projects. IT leaders are tasked with
the difficult challenge of taking business goals and finding a way, through technology, to achieve them in
a timeframe that meets business needs. But how do you help enable a better business?

One of the first things to look for is inefficient enterprise processes. By reducing the latency of these
processes, enterprises can free up personnel and capital and improve customer satisfaction and overall
competitiveness. The best way to shorten process cycle times is through enterprise integration. Yet, for
many organizations, enterprise integration is still too hard for IT and too slow for business.

Without good integration practices, it’s still too difficult for IT groups to respond quickly enough to
integration requests, process changes or meet new application requirements. For many organizations,
agile and effective integration remains a wish, not a repeatable practice. That’s why the combination of
modern event-driven ESBs and integration patterns can have a huge and strategic impact. A proven
ESB infrastructure will provide an organization with the foundation for IT excellence. But by using
integration patterns in conjunction with ESBs, IT managers will be able to change the economics of
integration once and for all.

2 ebizQ copyright 2007

The Value of an ESB

No one likes to throw away something that’s already working—especially when there are new
technologies that make existing technologies work even better. That’s why so many organizations are
starting to leverage ESBs as the foundation for a distributed event-driven integration backbone. An ESB
is software infrastructure that simplifies the integration and flexible reuse of new and existing business
components within a distributed service-oriented architecture. The ESB provides a dependable and
scalable infrastructure that connects disparate applications and IT resources, mediates their
incompatibilities, and orchestrates the interaction of components into composite services that are
themselves broadly available as services for reuse.

The primary benefit of using an ESB is its ability to decouple business components (events, services
and processes) so they can be developed, tested, deployed, scaled, connected, orchestrated and
managed separately. This decoupling allows integration teams to then easily extend or reassemble
components or build new composite applications and to test, deploy and scale those new behaviors with
little to no impact on existing running components. The backbone effectively provides component
isolation and scaling so those factors don’t slow the integration team or force downtime or major
scheduling delays.

With an ESB, initial integration projects can be easily built using a small number of components,
containers and brokers. Subsequent projects can incrementally expand upon that project, extending the
number of servers to build a more distributed bus that can extend the reach of the events, services and
processes of the first project to new applications or organizations. This ability of an ESB to be
incrementally deployed also reduces the project risk and lowers the upfront investment required. Over
time, as components are added to the bus, they form a pool of high-quality, tested, reusable building
blocks for future projects that can lower the risk to and increase the overall value of this type of
infrastructure. The impact is that each new project results in an enhancement and expansion of the
scope of the ESB in support of the business, enabling incremental alignment between business and IT
goals.

When you consider an ESB’s flexible support for the integration life cycle with its ability to be deployed
incrementally, and then combine that with a team with a growing arsenal of reusable components,
experiences and knowledge of the business, you can make integration a competitive or strategic
business advantage.

The Role of Integration Patterns

Everyone is familiar with the notion of “plays” in sports. As a football team prepares for a game, the
plays are essentially a pre-defined set of moves that are known to work well in certain offensive (or
defensive) situations. By combining these patterns with the experience of the players, a team can put
together an effective game strategy. It’s the same with integration contests.

By combining business components (events, services and processes) in the right situation, an IT
organization and its integration team can create effective and efficient ESB-based integrations without
re-inventing the wheel each time. The set of integration components that can be reused to solve
commonly occurring problems grows over time – expanding the repertoire of integration moves available
to the team.

An important aspect of integration patterns is that they’re not static, “out-of-the-box” solutions. Rather,
they can be changed by the team and recombined on-demand to meet new and even more complex
integration needs.

3 ebizQ copyright 2007

Integration patterns enable organizations to maximize the effectiveness of their integration projects to
deliver real business improvement. Consider the following benefits of integration patterns:

■ Fosters team communications. Integration patterns foster the development of an integration

language that encourages teams to “reuse the forest, not just the trees.”

■ Accelerates solution delivery. Integration patterns reuse existing business components,

allowing the integration team to rapidly assemble solutions in less time.

■ Ensures quality execution. The high degree of component reuse permitted with integration
patterns ensures that development resources can focus their limited resources on the new
aspects.

Integration patterns are a worthwhile addition to any integration project because of the efficiency gains
they bring to the development process. The combination of integration patterns and ESBs takes enter-
prise integration to a new level in the strategic business technology category. As the next section illus-
trates, companies are finding that the benefits of combining the two provide almost immediate positive
pay-off.

Common Integration Patterns

The easiest way to put integration patterns and ESBs into context is to take a closer look at some of the
most common patterns that exist in most integration projects, and see how their real-life use has
brought measurable results to some representative enterprises. This section will highlight four macro
integration patterns, although there are many more lower-level patterns and infinite variations that
organizations can use as well. In addition, for each pattern, we’ll offer examples of firms that are
benefiting from using the ESB with this pattern today.

Continuous Pipeline Processing Pattern

■ Continuous Pipeline Processing Pattern - ESBs can be used in integration projects that are
trying to move from sequential batch processing to continuous event processing mode. This
situation can be seen across industries and throughout different enterprises. Whether it is the
need to reduce cycle times, reduce processing costs for peak transaction loads, or implement a
more flexible architecture for the business, moving to continuous processing is a common goal for
many organizations and a well known pattern for many integration projects. There are a number
of reasons why companies seek to initiate an integration project that will include continuous pro-
cessing. Here is a closer look at how this pattern recurs in most enterprises, and how
adopting an ESB with this integration pattern can lead to a variety of efficiency gains:

■ Reducing cycle times. In many organizations, the pace of business processes is

slowed by incompatible business applications that have been bridged with batch
processing or manual intervention. Each additional application that requires batch
processing can cause another delay to the process.

■ Improving data consistency. Batch processing also causes inconsistencies in the

data that different members of the organization are using. When people and applica-
tions are using data from different batch cycles, the results can be disastrous. It is diffi-
cult to find one version of the “truth,” and many business errors can occur.

4 ebizQ copyright 2007

 ■ Architectural flexibility. The desire for architectural
flexibility is quickly becoming a necessity for today’s
business operations. Through mergers and acquisitions,
companies are finding themselves with business
processes that span multiple systems that do not
interoperate. Often, there are applications from different
companies that have merged, each with its own
batch requirements, and it is challenging to try to merge
them into a single process chain. Similarly, trying to add
a new sales channel can bring multiple integration
challenges.
Using an ESB in a continuous processing integration
pattern can help these different situations by adding effi-
ciency to the integration process and solving many of the
common issues associated with batch processing. With
an ESB, the non-disruptive “leave-and-layer” approach
enables back-end systems to be integrated with event-
driven systems without coding or replacement costs.
Remote Information Access Pattern
■ Remote Information Access Pattern - Portals have
fast become the standard for enabling a variety of users
from within and outside the enterprise to gain access to
important data. Today’s portals often combine data and
content from a myriad of back-end systems, and often
they need to present the dynamic information in different
ways to different audiences. Portals work fine integrating
with a back-end data source that is local to its use. The
portal application gets more complicated when multiple
data sources need to be integrated and some of them are
either shared or not local to the portal application itself. In
these cases, data integration logic gets buried in the portal
application, forcing changes to the integration logic each
time a database or application change is needed.
Using an ESB in a remote information access pattern can
provide a flexible layer that will more easily provide the
access to the information in a way that meets business
goals. Using an ESB layer with a remote data access
integration pattern can also make the task of quickly
connecting applications and systems through portals to
new customers easier. Once the first integration has been
accomplished with the portal and back-end systems, each
5 ebizQ copyright 2007
Continuous Pipeline Processing Pattern
Real-world Business Example
Example 1: A major home medical provider
used an ESB with a continuous processing
integration pattern to improve order automa-
tion and real-time customer views. The com-
pany eliminated the 15-day transaction lag
time it was experiencing in order to meet
wholesale customer service level agree-
ments.
Example 2: A very large European food
retailer needed to streamline its order/fulfill-
ment process across stores, distribution and
suppliers. Using an ESB with a continuous
processing integration pattern, it reduced its
replenishment cycle from three days down to
two, saving $15 million a year in reduced
inventory carry costs.
Remote Information Access Pattern:
Real-world Business Example
Example 1: One local government agency
used an ESB platform with the remote infor-
mation access integration pattern to create
one portal with a single login across multiple
municipalities. The police, court, sheriff and
district attorney’s offices are all linked
together, and departments are able to pool
resources while protecting privacy, a critical
requirement for government agencies.
Example 2: After rapid growth through
mergers and acquisitions, an Italian financial
services company needed to integrate multi-
ple channels, including e-banking, to lever-
age sales and service organizations across
lines of business. Using the ESB with this
integration pattern integrated home banking,
telephone (IVR) and call centers across dis-
parate internal and external partner back-
end systems.
 subsequent connection becomes incrementally faster.
The results are an enterprise that can more quickly lever- Remote Data Distribution Pattern:
age portals for business advantage. A variation of this Real-world Business Example
same pattern also occurs when service companies expose
back-end integration logic as a web service to clients in
software-as-a-service (SaaS) applications.

Example 1: A leading U.S. financial servic-

Remote Data Distribution
■ Remote Data Distribution - In today’s distributed
enterprises, business is often being conducted across
continents and time zones. When there is one primary
source of data or information, the update schedule for that
information flows out to all other systems in a batch-
oriented (and, therefore, delayed) mode. This can cause
significant business challenges, including pricing errors,
product information errors, and other similar business-
critical issues. In a large enterprise, there are often many
of these back-office systems that run the business and
are vital to day-to-day operations.
Using an ESB platform to provide continuous data feeds
as real-time events to all connected locations is an ideal
way to solve the problem. With continuous feeds, some-
thing as critical as new product pricing can be disseminat-
ed to all locations simultaneously, enabling businesses to
operate in sync, no matter where their offices are located.
Using this integration pattern for remote data distribution
can help IT become more responsive to business needs
for real-time access to critical business information and
create a repeatable process for when new locations need
access to the information, or when there are new systems
to which access is needed.
es organization was suffering from pricing
errors and routing failures due to out-of-date
security master data. Using an ESB and this
integration pattern, the company was able to
achieve real-time distribution of security
master data across functions and geogra-
phies of the bank. The continuous data
feeds support intra-day risk assessment that
is necessary for Sarbanes-Oxley compli-
ance.
Example 2: A large German retailer had
pricing challenges among its various retail
locations and headquarters, making it diffi-
cult to react to competitive pressures. The
company used an ESB platform with this
integration pattern to send updated point-of-
sale reports to headquarters and price
changes to the stores. By making this hap-
pen in near real-time, the store-by-store
product pricing updates are now able to sup-
port advanced discounting strategies.

Response to Real-time Events

■ Response to Real-time Events - Business sometimes needs to operate at the speed of light.
Whether it is an emergency management system, a major airport, or a financial services organization
that monitors the stock exchange, many businesses must have ways of capturing real-time events
and notifying all necessary applications and systems in the network of the event. Integration is an

6 ebizQ copyright 2007

 obvious solution to this need, but it can be challenging to
use traditional, custom-coded, point-to-point integrations, Response to Real-time Events Pattern:
especially when the network consists of hundreds of sepa- Real-world Business Example
rate locations (and sometimes separate businesses) that
have their own security, applications and disparate IT
infrastructures.

An ESB can be used as an integration layer for this pattern,

enabling the automated delivery of real-time events, and
allowing new systems and applications to be added to the
network without having to interrupt the present operations. Example 1: A major East Coast city used
Because this integration scenario can be found frequently, an ESB with the integration pattern to con-
using an ESB and this scenario together enables the nect multiple emergency response centers
in different counties that need to share infor-
organization to quickly and efficiently manage events and mation during emergencies. The different
promote an immediate response to those business-critical locations had incompatible emergency man-
events, adding systems without disrupting business. agement system applications. Using an ESB
platform and this integration pattern, the city
was able to make these communications
continuously available to deliver real-time
emergency response events across the
A Future Vision: The Responsive Organization region. Now, new centers can join the net-
work without compromising their security or
Where will this improved enterprise integration lead? What their operational autonomy.
are the ultimate benefits beyond today’s tactical gains from
combining integration patterns with ESBs? A large European airport operator used an
ESB for its event backbone in a new termi-
nal. In this capacity the ESB promotes real-
The answer lies in the transformation of today’s businesses time event flow in support of 35 million pas-
into responsive organizations. Once organizations deploy an sengers a year through the airport and their
ESB and begin to use some of the patterns that have been interaction with baggage handling, passen-
highlighted here, there are many new opportunities available ger information, retail and flight information
systems. Hundreds of applications related to
to them. Over time, the patterns of integration will begin to
airport operations receive events from the
overlap, enabling and essentially exposing a new IT paradigm ESB in real time, ensuring the constant flow
for building and maintaining new enterprise processes. This of information within the terminal.
can be referred to as a responsive organization.

The responsive organization will build new enterprise processes that just weren’t possible in the past. IT
members will be more able to leverage their amassed skills in integration for rapid development and
deployment of functionality. This rapid development will lead to an even closer synchronicity between
business objectives and IT deliverables. Business change will invoke faster response by IT and,
therefore, bring better results. The entire enterprise will be more agile and be able to respond more
quickly to competitive pressures.

In the responsive enterprise, CIOs will be able to promote the ESB platform approach, replacing the
traditional software development with process assembly. These processes will be triggered by
enterprise events, be composed of reusable services, and communicate data in enterprise common
data formats. The process assembly will leverage the repeatable patterns that are discussed in this
white paper and made possible by out-of-box packaged solutions like an ESB. The effect on IT will be to
lessen the coding effort that today consumes valuable time and cannot be reused. Instead, the ESB
platform will enable proven, repeatable patterns that are easily identifiable in most integration scenarios,
and achievable with minimal additional coding (additional to the growing set of reusable assets
mentioned earlier). The results support rapid business change that can be seen directly in the bottom
line. As a result, CIOs and IT leaders will cement their position in the enterprise as catalysts for
business change.

7 ebizQ copyright 2007

Conclusion
The relationship between business strategy and technology application is increasingly becoming tightly
intertwined. CIOs and IT leaders are assuming more responsibility in delivering on business goals in a
cost effective and timely fashion. Because of the need to leverage existing technology infrastructure
while at the same time working toward a more agile, responsive enterprise, integration is a key focus
area for many IT organizations.

One of the most effective ways that IT leaders are addressing integration challenges is with ESBs. By
using ESB platforms as the event backbone of the enterprise, these companies are able to leverage
their existing IT assets while building a new, more flexible architecture for the agile enterprise. Adding
ESBs to an integration project will deliver higher levels of improvement than traditional integration
methods and provide strategic advantages to the enterprise.

For even greater results, by combining ESBs and common integration patterns, enterprises have found
the most effective way of leveraging their integration competencies. With ESBs, IT organizations are
able to build repeatable processes for common integration patterns, bringing unprecedented efficiencies
to enterprise integration. Using an ESB solution in conjunction with integration patterns will eliminate
custom coding, avoid unnecessary overuse of precious IT resources and enable the achievement of
business goals. Ultimately, with an ESB and integration patterns together, the entire enterprise will move
toward a more event-driven, flexible, responsive business that can react to changing business
conditions and be more responsive to changing business goals.

CIOs and IT leaders are in a unique position today. They have the ability to translate the newest
technologies into competitive advantage. Organizations that are able to identify, within their integration,
one or more of the patterns discussed in this report, should take a closer look at how to incorporate
an ESB platform that supports these common integration patterns. The results are likely to be far better
than the current process for integrating multiple systems. In most cases, they will mean the difference
between an enterprise that keeps up and an organization that has the potential to leap ahead.

Written by:
By Dave Kelly, ebizQ Analyst
Co-authored by Progress Software

8 ebizQ copyright 2007