Вы находитесь на странице: 1из 12

WHITE PAPER

Managing Data within Billing, Mediation and Rating Systems

Managing Data within Billing, Mediation and Rating Systems M a k e r s o f
M a k e r s o f B e r k e l e
M a k e r s
o f
B e r k e l e y
D B

Introduction

Building high performance production systems for billing and mediation relies heavily on making the right architectural choices needed to achieve the high availability and system throughput demanded by such systems. This paper examines some of the common key functional areas of these processing engines with a particular focus on data management strategies. We use the Berkeley DB database engine, an open source devel- oper database used in many carrier grade applications, to illustrate how data management strategies can be implemented.

The range of requirements is broad and there are specific computing challenges involved in the process of collect- ing usage statistics, applying a rate to these statistics and producing a useful subscriber bill at the end. This paper examines the global requirements which apply at all steps in the process and the specific requirements which apply to mediation/aggregation, rating and billing respectively.

22 MANAGING DATA WITHIN BILLING, MEDIATION AND RATING SYSTEMS

Contents

Introduction

1

Global Requirements

3

Data Throughput

3

Real-Time Data

3

Data Durability

3

Data Availability

3

Module-Secific Data Requirements

3

Reference Data

3

Live Data

3

Critical Design Considerations

5

Predictable Data Access

5

Simple Data Types

5

Service Orientation

5

Repetitive Access

5

Live Data Management Strategies

5

In-Memory Data Storage

5

Client-Server Data Storage

6

Bespoke Flat FIle Data Storage

6

In-Process Data Storage

6

Berkeley DB

6

Database Performance Comparison

7

Aggregating/Mediating CDRs

7

Real-Time Processing of CDRs

8

Rating/Fraud Detection

9

Compiling Reports of CDRs

9

Exporting Data

10

Deployment Considerations

10

Data Replication and Load Balancing

10

Data High Availability

11

Data Security

11

Database Maintenance

11

Summary

11

Global Requirements

Billing and Mediation systems are a key element in any operator’s OSS. They are often considered the most mission-critical components of the OSS, as outages or data loss have a direct impact on operator revenue. Any application must fulfill the following key requirements:

Data Throughput The proliferation of new services, new billing methods and new subscribers continues to drive data volumes higher throughout the system. Subscribers number in the millions and billable events are of the order of tens or hundreds of millions, depending on the billing cycle.

Real-Time Data Support for pre-paid services in particular requires that full end-to-end processing of billable services is possible in real time. Latency within a system architecture is often a limiting factor on meeting real-time requirements. System latency can be more difficult to address than system throughput as it is often a feature of the software architecture and cannot be efficiently remedied by the availability of additional hardware.

Data Durability Data loss equals revenue loss for an operator. Systems simply cannot afford to drop data under any conditions.

Data Availability Billing is a real time operation (particularly when sup- porting pre-paid services) and must always be avail- able. Billing requires rating information which, in turn, requires usage data. Any failure of any part of the sys- tem (e.g., rating engine) will result in failure of a whole system or a failure to achieve the operator’s desired level of revenue assurance.

WHITE PAPER

33

Module-Specific Data Requirements

In each part of the process, there are distinct problem types. The problem types can be broadly divided into two categories; reference data management and live data management.

Reference Data

Reference data relating to the operational or business knowledge of an organization – such as subscriber details, account histories or plans – is generally accessed infrequently and is of a relatively complex nature. Examples of reference data applications are rate plan management and subscriber relationship management. The frequency of changes/writes is relatively low (< 10,000-1,000,000 per day), and there are a high number of queries/data reads relative to data writes. Additionally, queries are more likely to be of an ad-hoc nature, and the associated database schema is generally more complex.

The storage of reference data is usually well understood and well suited to database management systems with support for high level schema definition and ad-hoc queries. Often, the choice of database is dictated by the demands of specific operators, who may have existing software or who rely on SQL as the common denominator between applications for integration and interoperability.

Live Data

Live data is data that has to be processed before it can become useful business knowledge or reference data. Using a call data record (CDR) as an example, it is of little use to the business without significant processing. Once it has been collected, rated and presented as part of a subscriber bill, it has distinctly more business value. Live data is also vital business data that is accessed very frequently in the course of the system operation, such as subscriber information caches/lookups, pre-paid bal- ance management, event/audit logs. By its nature, live data exists in large amounts since it is the lifeblood of the business.

44 MANAGING DATA WITHIN BILLING, MEDIATION AND RATING SYSTEMS

Examples of live data applications are usage collection/ mediation, rating and account state management. The frequency of changes/writes is high (>1 million per day), and there is a comparatively high ratio of data writes to queries. It is also generally the case that there is a small set of defined access methods/queries on the data, and these are usually known at design time. The correspond- ing database schema is often quite simple.

������������������
������������������
������������������������
������������������
������������
��������������
�����
�����
����������
����������
������������������������
����������
����������
�����������
�����������
������������������������
����������
��������
��������
������������
������������
������
������
������������������
������������������
������������������������
�����
�����
����������
����������
������
������������
������������
�������������
�������������
������
����������
���������
���������
������������������
������������������
��������������������������
������
����������������
Sample architecture for mediation/rating billing system

Critical Design Considerations

In order to build robust and scalable systems, it is clearly important to identify the weakest parts of the architec- ture. Handling live data places the greatest load in terms of data volumes and and greatest exposure to critical latency limits. For this reason, this paper focuses on data management strategies for handling live data.

There are certain characteristics of live data and the processing which must be carried out on it which can be exploited to deliver optimal performance:

Predictable Data Access

This is an excellent characteristic for optimizing sys- tem performance. Data can be stored and retrieved using database indices which are based on the most common access mode for the data. This enables expensive database join and query operations to be avoided. As these can involve tens or hundreds of times more processing than simple index lookup, this represents a huge potential efficiency improvement.

Simple Data Types

The data types associated with billing and mediation (for example, CDRs) are often structurally simple and do not have complex relationships that need to be specified within a high level data schema. The ability to dispense with complex schema means that expen- sive schema can be avoided in both the application and the underlying database.

Service Orientation

Many development organizations are adopting a service oriented approach to their systems devel- opment. This approach is characterized by encap- sulation of functionality within functional modules (“services”) whose interfaces are well defined. The traditional data-oriented approach, which is implied

WHITE PAPER

55

by relational systems, is in opposition to service ori- ented development, since it brings with it the notion that there is a separate data storage format and query language which service modules must supp- port. This adds additional complexity and, critically, performance overhead to a service-oriented appli- cation which is built to use a relational database system.

Repetitive Access

At high loads, billing and mediation systems make almost continuous use of the underlying data store as they write UDRs (Usage Data Records), audit logs and other data to ensure it is safe in the event of system failure. Any form of access latency, such as network/socket latency, will have a significant effect on performance when multiplied over the total num- ber of writes/queries.

Live Data Management Strategies

In-Memory Data Storage

Ideally, all handling of live data would be done in main memory, since this enables data to be stored and retrieved at optimal speed. Storage of CDR data in main memory alone is generally not appropriate, however, because the risk of data loss is unacceptable. Attention must be given to any form of internal data buffering or queuing which will be at risk if the application crashes. Consequently, all queues/buffers need to be backed up to some non-volatile storage mechanism. The options are outlined below.

6

MANAGING DATA WITHIN BILLING, MEDIATION AND RATING SYSTEMS

Client-Server Data Storage

A common storage solution is the use of a client-server, relational database management system (RDBMS). Often this may already be in use for the storage of refer- ence data elsewhere in the OSS, and so it is tempting to assume that it will be appropriate for storage of data in all situations. In fact, RDBMS are very generic storage solutions and bring with them many layers of functional- ity which are redundant when handling live data. Trying to improve system performance for the high volume of writes/queries involved in billing and mediation process- ing can involve a significant amount of development work to try and raise the throughput to acceptable levels. Examples of this are the batching of reads/writes to the database to reduce the effect of network latency and connection pooling used to parallelize access to the database. In extreme cases, a large in-memory cache is used in front of the database to improve response time. While this improves read times it does little for performance when faced with the high proportion of database writes that are associated with live data. The remote nature of a client server database places it at a distinct disad- vantage in ensuring rapid response and low latency to system transactions which must be durable.

Bespoke Flat File Data Storage

To improve performance, developers often build inter- nally-developed file system solutions to store data. In general, these solutions solve the problem of perfor- mance but often lack transaction support for full ACID transactions, recoverability and advanced high-avail- ability features such as clustering. Additionally, this approach diverts valuable development time from build- ing other features within the product. It can also neces- sitate a cost for porting to new hardware, as well as add on-going maintenance costs to the solution as all of the code associated with the internally-developed solution must be maintained and upgraded whenever the appli- cation is upgraded.

In-Process Data Storage

The ideal storage solution for building massively scalable billing and mediation solutions is one which combines the speed of in-memory access with the safety of trans- actional, secure disk storage. An in-process database

of this kind offers an elegant solution to the problems associated with handling live data. By removing the query latency, read performance and - more importantly - write performance can be dramatically improved over that offered by a client-server database management system. In addition, an in-process database can provide transactional and recoverability features often lacking in bespoke flat file solutions. Berkeley DB is an example of an in-process database commonly used in a wide range of carrier grade applications such as OSS, messaging services and network infrastructure, and it will be used to illustrate some of the optimization techniques for han- dling live data throughout this paper.

Berkeley DB

Berkeley DB is an open source developer database which operates in-process after being linked into the application. All data is stored as a collection of key/ value pairs within the database, where the key is the equivalent of an index, and the data is equivalent to a record in a relational database. Keys are typically ordered in a b-tree structure (although they can be structured in other ways), and many sets of keys can be used to store multiple views/indices to the same set of data. All data is stored in the native format of the controlling application, which eliminates a tremendous amount of processing overhead associated with relational storage systems.

Berkeley DB supports concurrent access, transactions, replication and high availability/clustering via C, C++, and Java APIs, as well as a variety of scripting languages.

WHITE PAPER

7

Database Performance Comparison

To understand the magnitude of performance gains available through using Berkeley DB, it is important to understand its relative strengths. In a recent simple benchmarking exercise, a small data record of a similar size to a CDR (~60-80 bytes) was written and read using three database configurations:

1. A C++ application using Berkeley DB 4.2

2. A Java application using Berkeley DB 1.5 Java Edition (JE)

3. A Java application connecting to a relational data- base management system (in this case, MySQL 4.0.21 accessed via JDBC).

This simple test measured the time taken to write a fixed number of records to the database, one at a time. The read test measured the time taken to perform a single query of the database, which returned a given number of events (records). The query was a simple indexed lookup on a single column of data. No attempt was made, however, to optimize or batch queries to try and avoid network/query latency as would typically be done to improve the perfor- mance of a client server database.

The test was performed on a notebook PC running Linux (Specification: Intel Centrino 1.7MHz, 512Mb RAM).

Sample of record stored:

Seq Source Destination EventType Description

1 142.3.1.5 612.3.41.5 16

Link up

Write Time (ms)

#

Records

MySQL

Berkeley DB JE

Berkeley DB C++

 

1000

1168

492

10

 

2000

2261

680

20

 

5000

5673

1103

60

 

7000

7156

1396

90

 

10000

10167

1905

130

 

100000

100787

14624

1880

 

1000000

   

3167

Query Time (ms)

 

#

Records

MySQL

Berkeley DB JE

Berkeley DB C++ *

 

1000

9

1

0

 

2000

10

2

0

 

5000

15

5

0

 

7000

17

8

10

 

10000

20

14

10

 

100000

142

138

80

 

1000000

   

800

Comparative time to write data and perform a single query.

As can be clearly seen, the level of performance - particular-

ly for writes - far outstrips conventional relational database

systems. The response of adding an in-memory cache to

a relational database offers little relief here, since perfor- mance depends heavily on the performance of database writes, which must necessarily involve disk access.

Aggregating/Mediating CDRs

In some instances, CDR logs need to be aggregated to produce usable information for billing purposes. As an

example, consider a call log that makes entries for the start and end of a call/session that then need to be correlated in order to make complete call records. Typically, the call logs are imported into the database first to ensure they are per- sisted and then processed to find matching events. Using

a relational database, this requires several queries or joins

on the database tables to produce the result. With Berkeley

DB, indexing can dramatically improve correlation lookup times and eliminate entirely the need to perform joins on database tables.

*The accuracy of the system clock (clock_t) on the Linux x86 test machine was insufficient for measuring the time taken for small numbers of queries using Berkeley DB C++. Note that Berkeley DB offers other con- figurations (DB_TXN_NOSYNC, DB_TXN_WRITE_ NOSYNC) for even higher performance where relaxing durability is a possibility.

8

MANAGING DATA WITHIN BILLING, MEDIATION AND RATING SYSTEMS

Take an example CDR with the following fields:

LogId Date Time Trunk Number

Type

2345 25/12/04 12:01.01 1238523 00441234567 Call connected

2346 25/12/04 12:12.23 2356723 00441456789 Call connected

2347 25/12/04 12:14.06 1238523 00441234567 Call disconnected

2348 25/12/04 12:15.45 1238523 00353874107 Call connected

2349 25/12/04 12:34.12 2356723 00441456789 Call disconnected

To correlate the events, one might employ a join to find log events where the trunk and dialed number match each other and then pair the results to produce a single billing object per call. For example:

Date

Time Trunk

Number

Duration

25/12/04 12:01 1238523 00441234567 13:05

Berkeley DB uses a simple key-value model for indexing data. Typically, the key value will be an index value cor- responding to a column of a database table. Berkeley DB supports and manages multiple sets of keys on the same table of values. In this example, assume the unique primary key is the LogId. Additionally, we can use [Trunk + Number] as a secondary key/index to the data so that multiple entries can be entered using the same duplicate secondary key. By using a simple cursor to the data, all the entries for a particular [Trunk + Number] combination can be read very efficiently.

Real-Time Processing of CDRs

The greatest shakeup of billing, mediation and rating architectures has come from supporting pre-paid services. This places all validation, mediation and rating functions directly in the subscriber’s call path. The demand placed on the OSS to support this has made end-to-end system latency the key requirement for systems handling pre-paid services.

Berkeley DB’s exceptional performance and in-process architecture make it ideal for implementing subscriber cach- es, balance management databases and other forms of data

store where access speed, isolation and durability are para- mount. Berkeley DB supports a configurable cache for data storage and can also be configured to run entirely in-memo- ry if needed. It is routinely deployed as part of server groups using replication which process submitted events from a central event queue/buffer to perform balance updates or record mediation.

Berkeley DB supports a number of internal database struc- tures. The most common and general purpose of these is the b-tree. Berkeley DB also supports hash, queue and recno access methods internally. For real-time processing of CDRs, a recno/queue indexed database operates as a very efficient queue/buffer which is particularly suited to storing and retrieveing fixed length records. CDRs can be pushed into and popped out of the database which prevents data loss or out-of-memory problems which can occur when real time billing servers are subjected to “bursty” data traffic. Berkeley DB is frequently used as a high speed persistent buffer in data critical and high throughput applications (e.g., SMS/MMS messaging and email gateways) to make applications more resilient to high traffic loads and denial of service attacks.

Without persistent storage, the buffer would fill up, and then data must be dropped or the application will run out of memory. Neither case is appropriate for applications such as billing and mediation, where any data loss in unacceptable.

WHITE PAPER

9

Rating/Fraud Detection

Rating of billing objects can be a difficult process to get right given the ever-changing list of rate plans and services that can be applied to subscribers. Rate plans are becoming increasingly complex as more network technologies and billable services become available to subscribers. Rate plans are complicated and prone to significant structural revision from time to time. For these reasons, they are often stored in a relational (or potentially XML) database which provides an array of tools for managing this complexity. Rating manage- ment applications are also well suited to relational databases as the system performance is a secondary consideration to the requirement for ad-hoc query support and complex schema and relationship management capability.

The additional overhead of database schema process- ing layers make relational databases unsuited for the

timely bulk processing of CDRs. By contrast, Berkeley DB

is ideally suited to serve as the core processing engine

for performance-critical components of systems such as this. Many customers have found an optimal solu-

tion using a rating engine that retrieves rate plans from

a number of SQL databases, combined with core CDR

data processing using an in-process database such as Berkeley DB.

Date Time Subscriber Number Duration

Compiling Reports of CDRs

Compiling CDR information into useful per-subscriber billing data is another activity which requires intensive query and join operations on the underlying database. As with the CDR indexing example above, billing objects/ rated CDRs can be indexed to ensure fast retrieval for billing activities. For example, in exporting subscriber bills, the rated billing objects need to be selected for a given subscriber over a given billing period (say 30 days). Here, a secondary index is used which combines the subscriber number and the billing period.

Imagine all dates between 01/12/04 and 31/12/04 are identified as the same billing period, for example number 36. The secondary index would be [1238523 + 36] in this case. As with the CDR example, all entries for a sub- scriber in a given billing period would be immediately accessible without requiring a database join operation to be performed. A cursor is used to iterate over the entries associated with the secondary key to retrieve the com- plete subscriber billing information.

Rate

25/12/04 12:01

1238523

00441234567 13:05 0.45

27/12/04 15:17

1238523

00353234567 08:26

0.30

Our customers have also found that Berkeley DB is a good solution for implementing correlation engines which examine trends in usage statistics for fraud detec- tion and for building other value-added services having similar requirements for fast access and custom indexing of usage records.

10

MANAGING DATA WITHIN BILLING, MEDIATION AND RATING SYSTEMS

Exporting Data

Exporting processed information to external systems is the last step in the process. There are myriad ways to export data to external systems via middleware (Web services, SOAP, CORBA, MQ Series), direct database connections via SQL and file based transfer/import/export. Berkeley DB stores data in the application’s native format and does not require extraneous mapping prior to export. This optimizes the export process, preventing it from becoming the weak link in the processing cycle.

Deployment Considerations

Data Replication and Load Balancing

In order to achieve a five-nines level of availability, it is necessary to defend against system failure by duplicat- ing data in multiple locations. Berkeley DB supports data replication across multiple machines which affords seamless and automatic horizontal scaling to the appli- cation. Beneath the covers, Berkeley DB implements a single master replication model with a replicated envi- ronment. All replicas are available for reading (all writes are internally re-directed to the master database) within the environment.

Developers have complete control over the synchroniza- tion of replicas and transactions can complete when the write occurs at the master or when all replicas receive the update depending on the level of guarantee required.

�������� �������� ������ ������ ������
��������
��������
������
������
������
������
������
������
������
������
������
������
����������
������
������
������
������

����������

������ ����������

������������������������

���������� ���������� ������� �������
����������
����������
�������
�������

������������������������

���������� ���������� ������� �������
����������
����������
�������
�������

������������������������

���������� ���������� ������� �������
����������
����������
�������
�������

������

������

�������

�������

�������

�������

WHITE PAPER

11

Data High Availability

In a replicated environment, Berkeley DB replicas con- stantly monitor their connection to the master database. In the event of failure of the master or partitioning of the network, the replicas call a Paxos-compliant election process and a new master is chosen from N/2+1 of the interconnected replicas. Selection is based on the most recent log record and on a configurable priority scheme (e.g., it may be desirable to favor machines with faster disks or more RAM). This prevents a minority of replicas from electing a new master if they become isolated due to network partition.

Data Security

Securing data stored in a Berkeley DB database is easier than securing data stored in client-server databases (which must communicate between client and server using externally monitorable network traffic) as there is no requirement for SSL or other authentication protocols to be supported in the application. This is because the application and the database management code are linked directly together in a single process. Berkeley DB supports an encryption plug-in which encrypts the data directly on the disk, thereafter requiring a password from the application. In addition, Berkeley DB can restrict access to its in-memory cache to protect against unau- thorized access of system memory.

Database Maintenance

Berkeley DB does not require regular maintenance - or even a DBA - which greatly reduces the operational expense of systems deployed using it. For this reason, many of the best known names in the telecommunica- tions business depend on Berkeley DB. A partial list includes Cisco, Motorola, Amdocs, Ericsson, LogicaCMG, Alcatel, Tellabs, Openwave, Jabber, Hitachi, Lucent and AT&T. With over 200 million deployments, Berkeley DB is the natural storage solution wherever data is on the move – in the handset, at the switch, at the message center, in the OSS and in the billing system. Berkeley DB offers exceptional performance, zero maintenance deployments and unmatched reliability.

Summary

In the highly complex and demanding realm of media- tion and billing, there is no “one size fits all” solution for data management. Handling the volume of live data that would need to be processed immediately or refined to produce billing and subscriber information is often the critical weak point of a solution and a limiting factor for system scalability.

Reference data is well suited to traditional client-server relational databases. The ad-hoc query capabilities of SQL and wealth of tools available to users make them appropriate for managing this data. For live data, consisting of millions of records which are in constant motion through the billing system, other architectures are preferable. Berkeley DB is a vital component of these systems for many of the world’s leading operators and solutions providers.

12

MANAGING DATA WITHIN BILLING, MEDIATION AND RATING SYSTEMS

Sleepycat Software www.sleepycat.com makes Berkeley DB, the most widely used open source devel- oper database in the world with over 200 million deployments. Customers such as Amazon.com, AOL, British Telecom, Cisco Systems, EMC, Google, Hitachi, HP, Motorola, RSA Security, Sun Microsystems, TIBCO and Veritas also rely on Berkeley DB for fast, scalable, reliable and cost-effective data manage- ment for their mission-critical applications. Profitable since it was founded in 1996, Sleepycat is a privately held company with offices in California, Massachusetts and the United Kingdom.

For further information, please contact Sleepycat by sending email to info@sleepycat.com or visiting Sleepycat’s website at www.sleepycat.com

 

Corporate Headquarters

West Coast Office

European Office

Telephone

Sleepycat Software Inc. 118 Tower Road Lincoln, MA 01773 USA Sleepycat Software Inc. 5858 Horton
Sleepycat Software Inc. 118 Tower Road Lincoln, MA 01773 USA Sleepycat Software Inc. 5858 Horton

Sleepycat Software Inc. 118 Tower Road Lincoln, MA 01773 USA

Sleepycat Software Inc. 5858 Horton St. Suite 265 Emeryville, CA 94608 USA

Sleepycat Europe Ltd. Coronation House, Guildford Rd. Woking, GU22 7QD United Kingdom

+1-978-897-6487

+1-877-SLEEPYCAT

 

(Toll-free, USA only)

M a k e r s

o f

B e r k e l e y

D B

 

wp_billmed_0305