WHITE PAPER

Delivering Real-Time BI with

Complex Event Processing

Seth Grimes, Alta Plana

www.sybase.com
 INTRODUCTION
In today’s business climate, “time is money” has never been truer. The pace of business is fast, operations are 24x7,
physical borders have been erased, and the flow of information is constant. Economic pressures weigh on every
business. Competitiveness hinges on knowledge and on timeliness, on delivering the right information to the right
people and systems at the right time. A decision advantage — relevant, complete, accurate, and timely data — the
ability to see and act on opportunities and vulnerabilities before the competition does — is essential.

Traditional data management and analysis solutions fail to deliver this advantage. Relational databases were
designed for transaction processing, not speed-of-data decision making. Mainstream business intelligence similarly
can’t keep up with the variety of data sources — sensors, market data, in-process transactions, Web clickstreams, and
the like, many of them high volume — that affect business operations and customer experience today. BI solutions
that were designed for static, cleansed, historical data help you understand where your business has been. They don’t
provide for automated, real-time response to business opportunities and risks as they arise.

The gap between traditional BI and real-time analytical needs is where complex event processing (CEP) and the
notion of continuous intelligence — support for a now, cross-source, 360-degree view of the operating environment —
come into play. CEP technology first emerged in response to highly demanding, highly focused trading, intelligence,
and communications problems. Early, custom-coded applications were hard-wired for a narrow set of data sources and
processing algorithms. These applications have been supplanted by commercial offerings that integrate filtering and
analysis of high-volume data streams — market-feed data rates, for example, edge into millions of ticks per second —
with the ability to consume messages from diverse sources, supporting generalized, predictive analyses.

Commercial solutions include tools for both developers and end-user analysts. They utilize design studios, data
schemas, query languages and utilities, and dashboard displays that will be immediately familiar to BI practitioners.
The best of them allow businesses to embed front-line analytics in their operational systems. They build on BI
foundations to deliver decision advantage to the broad business-analytics market.

This white paper explores complex event processing and the extension of BI interfaces and applications to support
event-driven, real-time, operational BI and automated decision making: continuous intelligence.

REAL-TIME, RIGHT-TIME BUSINESS DECISION MAKING

Competitive pressures have created an analytics imperative, motivating adoption of business intelligence and data
warehousing technologies that seek to extract business value from data assets. Traditional BI and data warehousing
on their own, however, fail to cover the full set of modern-day operational needs. Their shortcomings have spurred
organizations to move beyond BI to complex-event processing.

Traditional BI is:

• Historically focused rather than “operational” and predictive.

• Designed only for monitoring, reporting, and interactive analysis rather than embedded, event-driven decision making.
• Highly specialized, for instance for performance management and financial analysis, and not pervasive.

Similarly, the Load > Index > Query approach supported by traditional database management systems (DBMSes) and data
warehousing cannot keep up with today’s need for speed, for faster data and low-latency, “real-time” analytical processing.

Processes and Interfaces: BI versus CEP

CEP solutions build on traditional BI and data-warehousing techniques and interfaces, leveraging practitioners’
hard-won knowledge and experience. They enable a jump to real-time, right-time business decision making
via familiar, proven BI and data-warehousing analytical processes, extended to data streams, service oriented
architectures, and event-driven processing, as illustrated in Table 1.

1
 ANALYTICAL PROCESS
Data sourcing
Integration & cleansing
Computation of metrics & indicators
Output/results processing
BUSINESS INTELLIGENCE
database tables; ODBC/JDBC/
Analysis Services sources
ETL, ELT tools; in-database (data
warehouse) integration
SQL queries and functions; point-in-
time metric/KPI values
record sets; operational applications
must poll the database
COMPLEX EVENT PROCESSING
databases, data streams, messaging
systems, “event clouds”: via adapters
and service subscriptions
transformation and normalization of
live “data in flight”
continuous computations on live &
historical data via extended SQL
record sets and KPIs, output streams,
business rules engine integration

Table 1: Complex Event Processing extends BI analytical processes.

There’s more to analytics than processes however. We are also concerned with the user view, hence the CEP-BI
comparison offered in Table 2.

USER INTERACTION BUSINESS INTELLIGENCE COMPLEX EVENT PROCESSING

Interfaces dashboards, reports, pivot tables: event-enabled dashboards: backed by
point-in-time values continuous computations
Enterprise integration portals, developer kits, design studios portals, developer kits, design studios

Table 2: User facing/integration elements of BI and CEP.

Enterprise requirements do go beyond technical features and interfaces. To support the jump from business
intelligence to continuous intelligence — from static BI to continuous, event-driven analytics — enterprises need the
robustness, performance, scalability, security, and high availability delivered by enterprise-grade CEP systems.

Understanding CEP Processes

For maximum benefit, the two styles of analytics, BI and CEP, should be integrated into everyday business operations,
BI for strategic decision making and CEP for tactical monitoring and for speed-of-data action. With BI as a foundation,
we will look at CEP processes as they relate to operations.

Data sourcing
BI and CEP derive analytical value from data that affects business outcomes. BI’s aim typically relates to improving
organizational performance using transactional data extracted from corporate systems. CEP taps those transactional
sources, but as the data is produced as well as via database and data warehouse queries.

CEP accesses live data streams. Examples include time-stamped stock-market tick data with data tuples
representing the trade/offer/bid price and share volume for a given security; telephone-call records, credit-card and
bank transactions, and sales reported as they occur; Web-site interactions; and temperature, pressure, movement, and
other physical conditions reported by sensors. Adapters provide the interface between stream and message sources
and CEP engines.

Integration & cleansing

BI has long relied on extract-transform-load (ETL) for data integration and cleansing; extract-load-transform (ELT)
with in-database transformations and disciplines such as master data management (MDM) have become popular.
By contrast, CEP integration and cleansing are literally streamlined: the software filters and joins data from disparate
database and stream sources as part of the continuous computation process.

With traditional, BI-reliant data warehousing, data latency — a lag between data arrival and readiness for analysis —
occurs when values are written to disk and indexed for queries. BI’s use of batch updates and dependence on
conventional data structures further creates analysis latency. CEP reduces both classes of latency to the millisecond
level via in-memory processing and speed-of-data computations.

2
 Metrics & indicators
While BI algorithms can be pretty sophisticated, most computations involve aggregation (summing and counting)
and simple calculations such as percentages. Key performance indicators (KPIs) are derived performance-management
values that involve more complex calculations, and data mining adds a further sophistication by generating predictive
models from data. CEP supports BI-style computations and can also accommodate predictive models. Both are
extended with the ability to discern event patterns over sliding time and record windows to turn BI’s static, point-in-
time metrics and KPIs into continuously computed, action indicators.

Output/results processing
The use of BI KPIs, reports, and analyses rarely goes beyond dashboard monitoring. CEP enhances BI-style
monitoring and is also capable of driving automated, operational decision execution. In doing so, CEP responds to
events that reflect actions or data-defined conditions: a Web-site visitor adds an item to her shopping cart, the
inventory of product X falls below 50% of average weekly sales. The outcome is real-time business optimization,
complementing BI and its strategic focus.

BI IN REAL TIME… AND MORE

Decision-systems expert D.J. Power describes business intelligence as “a set of concepts and methods to improve
business decision making by using fact-based support systems.” Traditional BI tools, however, are hard-pressed to meet
Power’s definition in a world of always-on business. It takes CEP, building on the foundation of traditional BI, to keep
pace with emerging operational analytical needs.

Traditional BI suites rely on reporting, OLAP pivot table, charting, and dashboard interfaces to tease insights out
of historical data. Some vendors offer operational and real-time BI solutions in an attempt to break out of a reactive
straitjacket. Definitions and capabilities vary. Some define operational BI as simply reporting from operational databases.
This class of operational BI is limited to monitoring current indicator values. It supports, for instance, basic business
activity monitoring (BAM), but lacks capacity to integrate diverse sources and automate response to emerging
opportunities. Similarly, real-time BI often simply means real-time scoring of static predictive models backed by very
modest analytical capabilities rather than comprehensive ability to make sense of real-time operating conditions.

Data-mining and predictive modeling extend BI capabilities, albeit incompletely. Data mining discovers patterns
in data; the modeling exercise provides a tool for estimating future values of variables of interest or classifying new
cases according to past experiences.

Predictive analytics has applications for real-time credit scoring, financial-transaction fraud detection, compliance
monitoring, and the like, but the analyses remain static, based on predefined models and conditions. These functions
are forward-looking, but they’re still based on historical data. Predictive analytics and operational and real-time BI
are definite advances over traditional BI, but it takes CEP to go beyond what D.J. Power calls “pushing ‘real-time’
information to decision makers,” to supporting real-time, right-time business decision making, to progress from
business intelligence to continuous intelligence. CEP works alongside and established techniques, adding the time
and record windows, continuous computations, event-pattern matching, and operational embedding that move BI into
real time to deliver continuous intelligence.

3
Real-time data:
data stream and Complex Event Processing
messaging sources uous
Contin ce
e ll ig en
Int

Data Mining
tive
Predic
ics
An t
a ly
Interactive Analysis
(OLAP)
ss
Historic data: Busine ce
database and data Reporting
t e ll ig en
In
warehouse sources
Static query Dynamic query
and analysis and analysis
The path to continuous intelligence: static reporting > interactive analysis > predictive modeling >
predictive modeling > real-time, event-driven analytics

THE PATH TO CONTINUOUS INTELLIGENCE

The path to continuous intelligence starts with awareness of opportunity/risk gaps that stem from the inability of
traditional BI to respond to complex, event-driven business conditions. A responsive solution architecture, bridging
gaps, will cover both information technology and business processes. Proof-of-concept prototyping, an evaluation
and implementation best practice, demonstrates solution practicality and is followed by phased introduction of new
technologies and work practices.

Gap Assessment
Risk/opportunity gaps result from business scenarios that include:

• Failure to incorporate important business information, typically from untapped or underutilized data sources, in
decision-making processes.
• Unacceptable delay reacting to significant business events that can range from a sale or financial transaction to
an anomaly inferred from data patterns.
• Data analysis constraints imposed by standard SQL and data warehousing practices, BI reporting and analysis
interfaces, and traditional BI dashboards.

A gap assessment should include appraisal of current systems’ ability to respond to business drivers measured
against benefits obtainable via adoption of CEP.

Resource Inventory
A resource inventory will discover neglected or underutilized transactional, data-stream, and event sources —
clickstreams, credit-card purchases, service invocations, network packets, e-mail and IM traffic, and the like — that
relate to business drivers.

• Audit sources and determine their current and unrealized information value.
• Evaluate access, filtering, and transformation steps required to extract value.
• Understand how resources will be used in conjunction with data in databases. What outcomes are sought? What
interfaces — monitoring, interactive analysis, rules system, other — are required and how will they close gaps?

4
 Solution Design
A solution architecture should respond to both technical and business concerns. These concerns include leveraging
existing BI and data warehousing solutions and skill-sets to extend historical analyses and operational monitoring to
encompass live data and automated decision making and execution.

As part of the design process, it is important to:

• Include data sources and the analytics necessary to deliver desired outcomes.
• Account for usability, performance, scalability, reliability, and security.
• Support required data integration, transformation, and analysis steps; evaluate SOA as an enabling architectural
framework.
• Anticipate new data sources and analytical needs to ensure extendibility.

On the business architecture side, it is important to study the effectiveness of current business processes and the
potential for realignment. From there, one can see decision-automation possibilities and the best route toward CEP
integration into current or reworked business processes in interactive or embedded form.

SOLUTION SAMPLER
Complex event processing answers a broad set of business intelligence challenges. The common need is for
operational responsiveness, for event-enabled, low-latency analytics. Such solutions go beyond monitoring pre-defined
indicators. They deliver ability to act on business events as they occur.

For many vendors, operational BI simplistically means running reports off operational systems. They propose a
narrow definition of real-time BI as quick (within seconds) response to queries. These limited definitions suit some
businesses, but in a world of high transaction volumes, e-business, and sensor networks and monitoring devices —
given a flood information from financial market, news, Web 2.0 sources — these definitions fail to keep up with
evolving competitive imperatives.

We illustrate three common continuous-intelligence CEP applications.

Business Process Control

Conventional business activity monitoring (BAM) provides a dashboard interface for real-time display of static
performance indicators in graphical and numerical form. While BAM does tap into operational systems, typically
within a service-oriented architecture but as frequently via hard-wired connection, conventional systems are rigid and
quite limited in scope. CEP extends BAM in two directions. CEP provides an ability to monitor processes by drawing
data from multiple sources, detecting interesting and anomalous situations with trace-back to root causes. Second,
CEP solutions go beyond monitoring to drive actions in response to business events, helping organizations move from
BAM to business process control.

Capital Markets
Capital markets combine enormous competitive pressures due to the amount of money in play with extreme IT
requirements that stem from the volume of data generated and from traders’ reliance on sophisticated mathematical
models that discern opportunities and risks from price quotes, trading data, and company information. Data rates can
be extreme, into the hundreds of thousands of market “ticks” per second, with computations that apply time- and
record-windows to current data points, analyzed in conjunction with historical values.

Traders look for opportunities that stem from unequal information, and they apply what they find in automated,
algorithmic trading. Rapid execution equals competitive edge, an equation that applies in a increasingly set of
business domains.

5
Online Commerce
Conventional Web analytics, running off stored Web logs, does not respond to online business’s real-time
challenges. Typical reports and charts are not a real-time solution. CEP, by contrast, delivers an ability to act on
Web interactions as they are occur, going beyond BI and conventional Web analytics to deliver more responsive,
more profitable user experiences. CEP for e-business sees sequences of user actions as a form of data stream. CEP’s
combination of database queries and event-pattern matching, with continuous computations over time and record
windows, can reveal opportunities to better serve site visitors, boost sales conversions, reduce session-abandonment
rates, and increase satisfaction and profitability.

GETTING STARTED
A basic methodology — a series of proven, best-practices steps — is an invaluable guide for any organization in the
introduction of new technology, Complex Event Processing included. The following is an approach to extending a BI
program to encompass stream and messaging sources to support real-time, event-driven analytics.

Understand Business Drivers

It is essential to justify new technologies and processes in terms of business benefits, whether as a means of
pursuing opportunities, mitigating risks, enhancing competitive positioning and profitability, or all of these factors.
Several dimensions contribute to ROI: not just money saved or additional revenues earned, but also ability to do more,
more effectively. Measures vary by industry. For instance, a sales focused organization will aim to boost customer
capture and retention, while a credit-card processor would focus on reducing fraud. What are stakeholders seeking to
accomplish. How should achievement be measured?

By understanding business drivers, we can address business related objections, which most often focus on cost,
mission alignment, and concern about IT capacity and priorities, and win stakeholder buy-in.

Address Technical Concerns

Study of use cases — customer stories that explain how others have harnessed CEP — can suggest how solutions
can be adapted for an organization’s own use. They’re the basis for evaluation and implementation best practices, and
when coupled with proof-of-concept prototyping, can be invaluable in resolving technical concerns:

• Lack of understanding of streams, event clouds, messaging services, and the business value of the information
they contain.
• Means of addressing short-comings of entrenched approaches & technologies: conventional BI, data warehouses,
rigid computing architectures, and so on.
• The migration from historical-only data query to dynamic, event-driven analysis.
• Solution performance, robustness, and reliability and the quality of support.

A successful POC will demonstrate ability to deliver competitive advantage. It will help an organization create an
implementation roadmap that anticipates challenges.

Phased Introduction
A successful implementation strategy typically involves phased introduction of new technologies, information
sources, applications, and business processes. An organization that already has a strong BI/analytics program enjoys
a head start. It’s nonetheless prudent, when CEP is first being introduced, to target discrete, clearly defined business
problems where impact can be measured.

The overall solution will involve more than technology. It will account for staff responsibilities, skill-sets, and
capacity and include the testing, training, staff augmentation (with outside personnel if needed), and “burn-in” time
necessary to ensure success. A carefully planning CEP introduction and early successes help pave the way for broad
acceptance and adoption, turning stakeholders into advocates.

6
 ABOUT THE AUTHOR
White paper author Seth Grimes is an information technology analyst and analytics strategy consultant. He is
contributing editor at Intelligent Enterprise magazine, founding chair of the Text Analytics Summit, Data Warehousing
Institute (TDWI) instructor, and text analytics channel expert at the Business Intelligence Network. Seth founded
Washington DC-based Alta Plana Corporation in 1997. He consults, writes, and speaks on information-systems
strategy, data management and analysis systems, industry trends, and emerging analytical technologies. Seth can be
reached at grimes@altaplana.com, +1 301-270-0795.

Sybase, Inc.
Worldwide Headquarters
One Sybase Drive
Dublin, CA 94568-7902
U.S.A
1 800 8 sybase

Copyright © 2010 Sybase, an SAP Company. All rights reserved. Unpublished rights reserved under U.S. copyright laws.
Sybase and the Sybase logo are trademarks of Sybase, Inc. or its subsidiaries. ® indicates registration in the United States
of America. SAP and the SAP logo are the trademarks or registered trademarks of SAP AG in Germany and in several
www.sybase.com other countries. All other trademarks are the property of their respective owners. 11/10