The Importance of Advanced Network Monitoring in Changing Times

For the last 20 years, most small ILECs and rural LECs in North America have been dependent on the tandem companies to provide them with their source billing usage data. Its been a marriage of convenience. Rather than go through the expense of collecting their own interexchange (IXC) billing usage, why not have the large tandem company such as an RBOC, etc. supply billing tapes to the ILECs to enable the ILECs to turnaround and bill the IXCs and other exchange carriers for access to their local networks. This happily organized world is steadily crumbling under the weight of massive industry change. Today, non-Bellcore compliant switch technology is being deployed in both fixed and mobile networks. And this has opened cracks even gaping holes -- in the venerable Automatic Message Accounting (AMA) system that for so many years has kept telecom peace on the continent. Here to tell us about these developments is Darrell Merschak, National Sales Manager, from Carrier Management Systems Inc. , a small company with a large installed base of 120 small to mid-sized carriers. CMSIs flagship product, NAMS Firefly, mediates AMA records, collects/polls switches, and is now migrating its customers to a full-fledged SS7 analysis system that correlates and reconciles AMA billing data streams that are often missing critical call identification data. Dan Baker: Darrell, your companys history is a fascinating one because you started off as a telephone company and only then became a probe manufacturer and billing analytics supplier. Can you give us a quick background on the industry problem that caused you to migrate to SS7 solutions? Darrell Merschak: Yes, and thank you. Heres the issue, Dan. New switches from foreign countries are entering the market and some of those switches are not compliant with North American standards for the proper creation of call detail records. Even the rural ILECs, who you might think would be the last to catch the trend, are deploying foreign made switches in their networks because the prices are low in relation to the overwhelming need for the rural ILECs to deploy broadband technology and find a cheap way to counter competitive threats in their service territory. The growth of mobile/wireless is also contributing to the problem. Wireless providers are the lost stepchildren of inter-carrier billing. Many wireless carriers

dont even think Carrier Access Billing (CABS) affects them. However, wireless radio network traffic is often downloaded to wireline networks for backhaul and the tandems pick up that traffic and deliver it for termination. The trouble is that wireless traffic is harder to identify because once again -- its call record structure didnt grow out of North American CABS standards. So essentially when the tandem sees this traffic it says: Gee, this record looks strange to me. The CIC field is all zeros, and the calling party is not even included. I cant make sense of this call traffic, but Ill pass it along anyway. Its become so confusing, that the tandem companies, whose job it is to figure this out for the smaller operators, have thrown up their hands, and ironically some even remain in denial of the problem. As a consequence, today the large LECs or tandems have small operators sign Annex agreements which are a 12-page addendums to the 300-page interconnection agreements basically stating: You will now pay us a transit fee for your traffic, and from here on out we will create billing records for you only on a best efforts basis. Therefore, though the large LECs are having a hard time identifying wireless traffic, it is really the rural LECs and CLECs most adversely affected and who are caught without a reliable and accurate source of usage data for billing. Now I understand youre proposing ILECs deploy SS7 network probes to solve this problem. But why is that the best alternative? Why not simply turn on the old usage collection systems? DM: Well, if you turn on your AMA collectors after 15 years of inactivity, you suddenly find that its hard to do. The knowledge and expertise is gone, and its costly to retrain people with the appropriate skills. I once worked at a company that relied on the tandem company billing records for 12 years, and when they attempted to begin billing record creation using their own switch equipment I discovered that it was 16 months since the last time they updated their switches with the latest NPA NXXs from Telcordias LERG database. It was obvious to me in that period of time there are a ton of telephone number changes and you cant possibly create complete and valid billing records. As the industry evolves and matures, youre going to need to move to probe technology anyway because first theres simply going to be more foreign manufactured switch equipment deployed. Second, soft switch technology will be increasingly utilized and provided by a host of vendors with a myriad of proprietary machine code to serve the growing demands for IP packet telephone networks. Third, another major development that will accelerate this trend throughout the telecom industry is the wide spread use of bandwidth hungry consumer devices that

will operate on wireless network protocols such as LTE and WiMax which are pure IP packet technologies. As a result, many carriers are already struggling with the problem of how to monitor and maintain always-on networks working to satisfy the demands of multiple types of bandwidth hungry devices and services riding their networks. We believe the only viable option today is to deploy probe technology and have the probe stand at the signaling network level to act as a uniform source to monitor all network traffic and record every one of the multiple handshakes required before a voice call goes through. In short, it is a detailed and comprehensive record of precisely what the network actually did. The switches are communicating with each other so theres no mistake about it. In fact, for years the central office engineers took all that very robust SS7 data and used it to groom their networks. Moreover, probes are versatile enough to monitor both SS7 in TDM networks and SIP in IP packet networks at the same time. Therefore, they not only provide robust detailed data because its based on the network handshakes that actually occurred during the set up and transmission of a call, but they are also switch agnostic and provide a uniform source of network monitoring data irrespective of the types or combinations of legacy switches, soft switches, routers, brouters, or bridges deployed in the network.

How big an issue is this? classified for billing?

What percent of usage records cant be

DM: I dont have a precise number on the industry error gap because it varies tremendously by company and service area setting, but I do know some extreme examples such as the former e.spire Communications, whereby 42% of their traffic was phantom traffic -- traffic they couldnt properly identify. The fields were stripped so they didnt know who to bill. It was personally conveyed to me by the former Vice President of Rates and Regulation that e.spire was told: Dont worry, its a transitory traffic spike that will go away. Well, it didnt go away and e.spire was forced to file for Chapter 11 bankruptcy protection. Another issue is bypass, which NECA [National Exchange Carrier Association] calls access avoidance. In this situation you have toll-type traffic sneaking onto local or EAS (Extended Area Service) trunks disguised as local traffic. However, with SS7 data, you can reliably say, Oh no, that should be a toll call because you can point to an originating OCN code and a JIP (Jurisdicational Information Parameter). Moreover, there are often other SS7 fields that can be

examined which are either not spoofed correctly or inconsistently spoofed relative to the other SS7 fields and even some AMA fields. For example, depending on what networks are involved, you can often directly determine where calls originate from SS7 by examining the OPC (originating point code) data and therefore at the least youre getting an idea of the interconnect IDs. Many times if a field in the AMA record is stripped, it may still be available in SS7. Also SS7 has some additional fields that can be queried as surrogates for AMAs calling party number such as the charge number, the originating LRN, etc. Moreover, when you begin to correlate and compare the AMA or EMI switch generated record data to the SS7 data you notice some obvious differences. Regarding the obvious differences you only need to be as skillful as lets say the English novel crime investigator Dr. Watson to solve and reverse this type of phantom or bypass access avoidance revenue loss. With respect to the more complex cases you need to develop greater skill and become a Sherlock Holmes but this is possible in short time with the tools available now. DB: Who are your competitors in the market for SS7 analysis software? DM: Well if you insist, to name a few Agilent, Tekelec, Tektronix, Tekno Telecom, GL Communications are all in the market. Tekelec is the 800 pound gorilla now because they serve as the Service Transfer Point (STP) for the network. Agilent is also well-established, especially at the large LECs. In fact, Ive seen clauses in Annex Agreement contracts that stated if we disagree on billing exchange record accuracy or record jurisdictionalization, Agilent SS7 technology will be used to resolve the dispute. You can bet these probe manufacturers are eager to showcase how you can leverage their SS7 technology for revenue recovery but the hard truth is that SS7 probe technology has tended to work against the smaller players because they couldnt afford to own it. However, our NAMS product is far more affordable for the lower tier markets and yet we also serve the needs of tier 2 or even larger carriers, and this is software that a carrier owns from day one and they dont have to pay rent on. The beauty is it allows you to directly correlate an AMA or EMI record to an SS7 record. Naturally I am biased but I dont think anyone puts together a package that contains SS7 probe technology and a powerful analytic software tool that can correlate all this data to produce significant revenue recoveries and rapid ROIs for such an affordable price. Part of the reason is that we did our homework first as a telephone company years ago to develop significant expertise in this area, and perhaps the other reason is that we have always strived to control our costs and right size our products including streamlining probe production to fit carriers needs.

Moreover, with CMSIs NAMS system you dont even have to repair the damaged record if you dont want to. You can convert the exception SS7 records (which correlate to AMA error bucket records) to AMA or EMI format which allows any CABS billing company or system to directly upload and invoice. DB: In the last two years, the FCC has been pushing for the legislation for wider deployment of broadband services to rural America. Whats your take on that? DM: Allow me first to say that carriers should take time out and read the FCCs proposed NBP (National Broadband Plan) and the most recent FCC NPRM (Notice of Proposed Rule Making). It is definitely in the interest of the carriers to provide continuous input in the form of reply commentary to the FCC on this proposed regulation. In my opinion, broadband provisioning by the rural ILECs is a double-edged sword. You are essentially creating a packet information super highway into your network which is exactly what the major VoIP providers need to steal your subscribers. Also, as proposed by the new regulation, the original rate support mechanisms such as USF and ICC (inter-carrier compensation) will be diverted to support broadband and many of the traditional subsidies may be dramatically reduced or diminished and in time ultimately eliminated. Moreover, the battle continues before the FCC that a VOIP originated call crossing to the PSTN must pay a termination charge. Also, as I mentioned earlier equally worrisome is that IP usage will accelerate to astronomical levels in the near future as it will be driven by bandwidth hungry devices such as the iPhone based on LTE and WiMax protocols and also even IP video which are all pure IP packet network technologies. Also troubling is a remaining underlying network technology problem that as VoIP and other broadband IP based traffic moves across a TDM network most ILECs are not going to see that traffic and consequently they will be only able to sell large blocks of bandwidth consuming switch capacity at low flat rate prices.

If the ILECs dont win the FCC regulatory battle and also catch on and start monitoring their network traffic, understanding the nature of the traffic, and figuring out how to earn revenue on the traffic, their businesses will suffer considerable harm. The logical conclusion is that these carriers will become so financial weak that either they will need to declare Chapter 11 bankruptcy protection or become a target for business takeover by another carrier. I have already witnessed evidence

of this because as a national salesman I call on a lot of telecom companies and an increasing number are no longer in business. Some argue that this is just the travails of an ongoing telecom industry technology battle akin to a Darwin process of natural selection. Perhaps so, but I am sure you will agree the prospect of becoming extinct does not have much appeal. The good news from a technology standpoint is that carriers can protect themselves and defend their networks because the same probe technology can monitor SS7 TDM or SIP based IP networks and produce billing data CDRs (call detail records). There are also a few companies that suggest and point to DPI (deep packet inspection) as a way to look at the contents of the traffic, but the volumes are huge and I dont believe that all this extra data extract work is necessary for a carrier to obtain sufficient billed revenue compensation. All you really need to inspect is the SS7/SIP stream of that traffic sometimes called SPI (shallow packet inspection) and with that youll have enough information. Youll know that a communication or bandwidth window was open for a certain duration and you will be able to render a bill. In addition, using probe technology to your maximum advantage, you can understand the nature of the traffic how much is VoIP and from where it originates. So if an operator is not paying his freight, you can exercise your right to degrade their service when the network is congested and protect paying customer quality of service which has been guaranteed by you under SLA contract. You cannot block a call, but if network service is impaired, you can take care of your paying customers first using the network traffic monitoring products that exist today. Again, with respect to the fast approaching tidal wave of bandwidth demands fueled by bandwidth hungry consumer devices and flat rate all you can eat pricing, carriers will soon begin to realize that in fact network resources are increasingly scarce and repetitive or continuous capital outlays for new network switch upgrades is increasingly expensive or even cost prohibitive. Moreover, carriers will also realize that operating IP based networks generate a new distinct set of complications. Ironically, this may sound contrary to some recent carrier advertising campaigns encouraging flat rate no limit usage designed entirely with the objective of carving out market share. However, I believe in short time as bandwidth demand ramps up, carriers will be forced to abandon this marketing strategy as too dangerous producing low returns much like the gun fights in the old days of the Wild West. In closing permit me to be blunt. Irrespective of present regulatory vagaries or even the final direction of FCC regulation, every carrier is a duck on a fence post if they rely on exchange record or switch data alone for source billing data, or dont uniformly and aggressively monitor their network traffic to better understand the nature of the traffic and how to earn a buck on it. Today carriers must do this in

order to defend their networks against exchange record billing data errors and new telecom IP packet technologies. Unprotected networks absent probe equipment performing traffic monitoring will be easily overwhelmed and carriers will miss important opportunities to leverage their current and next generation network switch investments for incremental revenues and profits.

About Darrell Merschak darrell@nams.net Carrier Management Systems Inc. National Sales Manager 500 Chestnut St., Suite 500 Abilene, TX 79602 Darrell Merschak has worked in the telecom industry for 25 years in various capacities as a director of revenue, manager billing operations, revenue assurance manager, regulatory and rates manager, service cost manager, information database analyst, sales analyst for AT&T, SBC, Ameritech, MCI, Pathnet, Looking Glass Networks, Tekno Telecom, RTC. Darrell has spoke before the National Exchange Carrier Association, TeleStrategies, ANMTA, NAB. Darrell holds an MBA in Finance and Economics and MSIS course work from Loyola University of Chicago and a BA in Economics from Knox College.

Dan Baker is Editor in Chief for the Technology Research Institute of BOSS Billing and Operating Support Systems telecommunications magazine.