System Monitoring Best Practices

For

BCM 6.0 & 7.0

version 1.0
Contents
OVERVIEW ....................................................................................................................................................... 3
KEY AREAS TO MONITOR .............................................................................................................................. 4
Automatically triggered events/reports ................................................................................................................4
Daily manual check ................................................................................................................................................4
Weekly manual check.............................................................................................................................................5
Monthly manual check ...........................................................................................................................................5
General good practices ..........................................................................................................................................5
LOGS IN BCM 6.0 ............................................................................................................................................. 6
Generally: .................................................................................................................................................................6
CEM: .........................................................................................................................................................................6
Call Dispatcher .........................................................................................................................................................6
Media Routing Server: ..............................................................................................................................................6
Federation Bridge: ....................................................................................................................................................6
WEB: .........................................................................................................................................................................6
Connection Server: ...................................................................................................................................................6
H.323 bridge: ............................................................................................................................................................6
High Availability Controller: .......................................................................................................................................6
Data Collector: ..........................................................................................................................................................7
SIP bridge: ................................................................................................................................................................7
Detailed CEM errors for timeouts between CoS & CDT ...........................................................................................7
LOGS IN BCM 7.0 ............................................................................................................................................. 9
Error types ...............................................................................................................................................................9
Features ...................................................................................................................................................................9
Logging Categories and Locations ......................................................................................................................9
SOLUTIONS IN USE BY CUSTOMERS......................................................................................................... 11
Call Robot ...............................................................................................................................................................13
OVERVIEW

Once a BCM system is installed and running in live production, it is recommended to monitor your
system on a regular basis. This helps to ensure the system continues to run smoothly, and any
potential issues are identified early. Many problems are preventable, such as a server running out of
disk space. By proactively monitoring your system, you can prevent problems from escalating and
potentially causing system downtime.
This document is designed to highlight the key areas to monitor in your BCM environment, and
suggest some best practices.

These are just guidelines which each customer and partner has to then review and adapt for their own
environments. The named third party solutions are only examples, you may have other solutions for
the same purposes.

Please also see the infrastructure_guide.pdf.

KEY AREAS TO MONITOR

Note that the frequency of checking is only a suggestion

Automatically triggered events/reports

 Critical server elements: A server monitor program should be setup to monitor items such as
hard disk space, CPU usage and available memory on the production servers. There are
solutions available such as; MS Operation Manager, HP OpenView, Whatup or Nagios to
monitor the system components and create reports. Typically the partners have had an
established system for hardware monitoring which they adapt for the BCM servers.
 CPU usage, and available memory. These should create automatic triggers in the event of one
of these elements passing a specified limit, such as less that 10Gig hard disk space available.
 SNMP traps from alarm server: The alarm server, included in the BCM software package, can
be configured to send SNMP traps. For example with the Alarm Server you can send alerts to
some 3rd party software, so emails/SMS’s/SNMP alerts can be sent to notify you of any failure.
Other customers have implemented a “ping check” to constantly poll the virtual unit’s IP and
see if it is reachable.

SNMP (simple network management protocol) is part of the Internet Protocol Suite as defined by
Internet Engineering Task Force (IETF). SNMP is used in network management systems to monitor
network attached devices for conditions that warrant administrative attention.
Several SNMP management systems exist on the market and there is no particular preference for any
of them.

Basically SNMP defines the following message types:

- SNMP GET REQUEST
- SNMP SET REQUEST
- SNMP TRAP

SAP BCM sends only SNMP traps. SNMP GET and SET messages can be used for managing other
SAP BCM infrastructure devices supporting SNMP.SNMP management systems typically include
sophisticated functions, such as ability to:
- Filter the most relevant messages
- Send e-mail and SMS alerts initiated by events in the monitored system.
- Log events for further investigation.
- Export event data to files, sheets or syslog servers for reporting purposes.

 Network traffic monitoring. BCM software relies on stable network connections, so it is vital to
have network traffic monitoring in place. This could bring up an alarm if network connectivity
between the sites encounter severe delays or connection breaks. This is particularly important
when having two or more active sites.

Daily manual check

 IA – Open IA and check are the processes running, and are they running on correct server. IA
running on a large display visible in the office – this way you can easily monitor the current
status of all virtual units.
 Windows Event viewer
Weekly manual check
 SQL jobs – There are SQL jobs in the BCM system which are scheduled to run on a regular
basis. These include job which creates the reports.
 BCM Logs – Check no errors appear (see later section). After making any major configuration
changes, it is recommended to check the logs for the related components immediately after the
change, and also the following day – in addition to normal testing after the change.
 Reports – check that the BCM reports are created correctly, and that the Monitoring application
displays data

Monthly manual check

 Windows/SQL – checking for the latest Windows & SQL updates help keep your system
secure. Currently Microsoft releases updates on the second Tuesday of each month.
 BCM updates – Keeping BCM updated with the latest fixes is important. Critical fixes are
contained in hotfixes released when needed. Service Packs are released infrequently. They
contain a complete build. It is important to keep the Service Pack level up to date, as typically
hotfixes are only released for the latest Service Pack version.
 Check for updated BCM documentation on SAP Service Marketplace
 Reports – check that the BCM reports exist and contain data
 Windows Performance Monitor – log onto the server(s) to check CPU & memory usage, and
available hard disk space.
 On the SQL server, check the database and transaction log size, and available disk space.
Compare with size last month to plan ahead.
 Remove old files from the mail server. Old mails can be deleted from the mail server also by
setting an appropriate value to the parameter DeleteOldMailsAfterDays. See the System
Administration Guide for more information about application parameters.
 Remove old log files. Log file directories are defined during the installation. Define appropriate
paths and logging levels, and follow the amount of collected data regularly. The amount of data
depends on the system size, configuration and tasks it is used for.

General good practices

 Maintenance: Scheduled daily backups of at least CEM, CPM, VWU & WDU databases.
 BCM training – it is important that there are key staff who are trained on BCM. Whenever new
versions are released, it is a good idea to attend a partner training session to keep up to date
with the BCM software. Having a minimum of two BCM trained experts per system is a good
practice.
 Document any arising issues – e.g. agents might have the same issue, if recorded somewhere,
then can link up related issues, e.g. calls from a certain gateway have audio issues
 Good communication between teams. Having good communication between IT, telephony, and
users is essential so that any issues are reported in a timely manner, and that the correct
people are informed.
LOGS IN BCM 6.0

Check there are no errors in the logs (for example CEM log). Normally this would be a manual process
but could be done for example on a weekly basis, and using WV to highlight
“ERR>|error|WRN>|EXC>” for example makes this easier or a batch with command findstr /r /s "ERR>
error WRN> EXC> " "[VU_logs_folder]\*.*" makes this easier.

The "log grepping" could be automated with third party applications. We are not aware of what
monitoring tools other partners use to examine the logs themselves, and at the moment we do not
have ready templates. Creating templates which best suite each installation is a learning process, so
there will be some improvements for what is required/false positives occurring.

Generally:
ERR> , EXC> , WRN>

CEM:
ERR> , EXC> , WRN>, Failed

Call Dispatcher
ERR> , EXC> , WRN>

Media Routing Server:

ERR> , EXC> , WRN>, Terminating WCFPCore

Federation Bridge:
ERR> , EXC> , WRN>

WEB:
error, odbc error (ignore case)

Connection Server:
fail (ignore case. Will include words such as Fail, Failure, Failed)

H.323 bridge:
EXC>

High Availability Controller:

Inactive, Inactivating, "no such process"
Data Collector:
ERR>, "Failed to write events into database transaction buffer" (This would mean that either the place
where data collector is running is out of space, or it has no connection to the database and it’s buffer
has run out.)

SIP bridge:
(below are actual errors reported in SIP Bridge logs)
Received command to unknown session
Message handler failed
Couldn't parse message from
Caught unknown exception while handling message from
Unknown protocol
Socket create error
Socket bind error
getsockopt error
Socket WSAAsyncSelect error
Socket recv error
ReceiveAsync error:
OnConnectFailure:
OnReceive error:
ReceiveAsync error:
failed with error
failed. Reason=
DnsQuery failed getting SRV record for
failed. Error
send error
socket listen error
socket accept error
socket select error
disconnected with code
disconnected on receive timeout
can't send - not connected yet
SetSecurityOptions() failed.
Expired transaction

The WRN> is not usually critical, but can be included just to see how many appear.
We would advise just monitoring these to start with to see how many occurrences they produce.
Prior to implementing these in the tool, it would be good to manually check for example your last
hour’s or day’s worth of logs, incase these are already present. For example having an email channel
with wrong username can cause errors and exceptions.

Detailed CEM errors for timeouts between CoS & CDT

(i.e. the server and the agent’s workstation)
1) These lines occur when there are no messages from the client for about 30 seconds. If there are no
other messages coming the client should send at least a keep alive message. The client is set to
“paused” so no calls are sent. At this point, the CDT might show “Opening connection”
08:53:44.126 3284 ERR> LIBIPCServer : REC_Timeout : Timeout for received messages
(796B6A0E92E67148907C40A5FE3076AF) : ('i109',)

2) If between 30 seconds & 2 minutes, the connection recovers, the Opening Connection disappears:
08:31:47.063 3284 ERR> LIBIPCServer : REC_Timeout : Timeout for received messages
(8C32315692BD8F4A871C68E42F7B60A3) : ('i109',)
08:32:10.352 7616 REC> _CHANNEL_RECOVERY_ = {'_EVT': '_CHANNEL_RECOVERY_', '_ID':
'i109', '_SAP_ID': 'CONTROL', '_REP_ID': 'i109'}

3) If the connection between CoS & Client is still not reconnected after 2 minutes, the CEM stops the
current session, from it’s point of view. Note in green there is the affected extension number and user
id:
08:55:06.826 4420 INF> LIBIPCServer.OnLostChannel :
('796B6A0E92E67148907C40A5FE3076AF', 'i109', '796B6A0E92E67148907C40A5FE3076AF')
08:55:44.448 1976 INF> Closing login session due to timeout : ('1234', 'John Smith')
08:55:44.448 1976 INF> LIBIPCServer.CloseChannel : ('796B6A0E92E67148907C40A5FE3076AF',
'i109', '796B6A0E92E67148907C40A5FE3076AF')

4) If there is a network break over two minutes, from the user’s point of view the CDT will just show
opening connection until the connection is recovered, and then automatically recover.
08:25:31.438 6248 INF> {'CallType': 'In', 'ANumber': '987654321', '_SAP_ID': 'CALL_CONTROL',
'_CMD': '_UNKNOWN_REP', 'BNumber': '1234', '_EVT': 'Disconnect', 'CALL_ID': 'CI_101L', '_MGR':
'UI_MGR', '_CLS': 'UI', '_REP_ID': 'REP_20324', '_ID': 'BRW_20324', 'DATA': 'ERROR'} : ('REJ>',)

Possible Cause for these timeouts in the logs:

- CDT/atl is somehow terminated so that it does not log out properly.
- The workstation is so busy it does not send these in time. Then the session is probably restored
within 2 minutes.
- Workstation restart: new session should be imminent. In this case at least number 1234 did not
reconnect (to CEM, at least), so this might be a network problem
- Network problems. Then the session might be restored within 2 minutes.
LOGS IN BCM 7.0
(initial version)

In BCM 7.0, logs can be configured in versatile way to increase relevancy and avoid collecting
unnecessary files.
For example, it is possible to set logging level of a location or category to the debugging level as other
parts of code are logged in error level only.

In addition to BCM legacy log file format two new formats are added, SAP Generic Log File (GLF) and
List Log. SAP Generic Log File (GLF) format enables using SAP Log Viewer and SAP Solution
Manager for analyzing and managing logs.

Error types
Almost all modules will use the above logging mechanism (including also CEM, MRS, CD etc. since
7.0 SP1), file formats and configuration style.

Thus, all errors (in BCM-formatted logs) are in ERR> (EXC> should disappear and ERR> printed
instead. If you choose GLF-formatting to be used, it is error. Warnings are as WRN> (in BCM, and
warning in GLF).

Features
Configure logging function via Windows registry with a value starting Log followed by attributes
delimited with periods, such as
Log<ObjectType>.<ObjectIdentifier>.<Attribute>.

Modules collect information in the log files that are named with the syntax: <module-
name>_<computer-name>_<virtual-unit-name>_yyyymmdd[_hh][_nnn].log

Log files are collected in the folder that is defined during installation as Log File Directory of the Virtual
Unit installation variable. The default value is $VU_HOME$\logs.
Logging Categories and Locations
Using logging categories enable that, for example, database administrators only receive database
related log files, and network administrators follow the network logs.

Logging Categories and Locations

Using logging categories enable that, for example, database administrators only receive database
related log files, and network administrators follow the network logs.
Log Levels Level Log Writing Includes
always Messages that should always be printed,
such as process startup notifications.
error Unrecoverable error has occurred. Some
or all functions of the module are
inoperable or perform incorrectly. Often
indicates malfunctioning hardware, major
misconfiguration, or a bug in the product.
warning Warning messages indicate of problems
that are somehow recoverable or are in
less important functions. For example,
minor misconfigurations, performance
warnings, temporary inavailability of a
service.
info Informative messages about business level
or service level objects.
trace Messages to trace the execution of the
code in more detail. Mostly contains useful
information for developers only while
providing information about how execution
of the code occurred.
debug Most detailed messages about internal
execution logic of the code. Mostly
contains useful information for developers
only. Often used for tracing rare issues or
bugs in the product.

The global log level is the root level and cannot be assigned to be inherited. Log targets (log files,
console etc.) and logging modules (locations and categories) are by default set to inherit the global log
level, but they can also be assigned to their own level setting. Some logging modules may be a child
module to another logging module; the above parent filter is then the level filter of the parent logging
module. Threads are set by default, to inherited to follow other effective level filters, but they may also
be given their own thread-local log level filter setting. Combining the above level filters and settings,
one may control specific parts of the code, threads or types of events that produce log entries.

Example

A server module that prints debug-level log to file and error-level log to console. Also, entries from
LibIpc code location are narrowed down to error-level. The registry of the module (module's own
registry key) would have the following registry value names and value data (shown as
ValueName=ValueData):
LogLevel=debug
LogConsoleLevel=error
LogModule.LibIpc.Level=error
SOLUTIONS IN USE BY CUSTOMERS
One large German customer uses the WhatsUp program, although this just collect SNMP traps and
monitor active virtual IP's.

One Swedish customer uses an application called www.pingplotter.com that pings the VUs every 10
seconds. It sends me an email according to different rules set up. They also use Pingplotter for
monitoring (pinging) Internet, servers and WAN servers. By using to WAN (or LAN) you can see
historical graphs regarding ping times and jitter.

Solutions from BaseN & Noval have been used for monitoring the network infrastructure and to spot
possible problems with connections and/or quality. They can simulate also RTP traffic, and if
necessary even monitor the connections from certain sites/workstations.

One Finnish Partner has been using test robots (built by Goodsign, partly based on our previous
WSAM) which use ClientCore phone for making test calls from remote sites every x minutes and at
the same time test logon/logoff/simple query over http. They also measure the time taken so that if
there are suddenly bigger/increasing delays etc they can alert someone to have a look even before
“everything explodes”. With the test calls they can validate “the whole chain” i.e. a call from a
softphone thru gateway to a service pool or voicemail box.
One Finnish Partner uses an application called CastleRock. Two screenshots below show examples
of the interface & display.

CastleRock SNMP network view:

CastleRock SNMP SAP BCM servers view:

Call Robot
One option would be to use Call Robots to check service availability as experienced by phone users,
e.g. by utilizing the ClientCore interface of BCM. The robot could make automated calls with a given
pattern and confirm that the system responds as expected. If any anomalies were found, the robot
could generate an alert to notify the administrators.

In a more advanced scenario the robot could in addition to calling make simple test queries to the web
server to confirm that the web sites and databases respond without unexpected delays.
The design of the automated robots can vary depending on what you want to monitor and measure.
An important factor is not to create too much loads by the testing.
© Copyright 2012 SAP AG. All rights reserved.
No part of this publication may be reproduced
ortransmitted in any form or for any purpose without the
express permission of SAP AG. The information contained
herein may be changed without prior notice.
Some software products marketed by SAP AG and its
distributors contain proprietary software components of
other software vendors.
Microsoft, Windows, Outlook, and PowerPoint are
registered trademarks of Microsoft Corporation.
IBM, DB2, DB2 Universal Database, OS/2, Parallel
Sysplex, MVS/ESA, AIX, S/390, AS/400, OS/390,
OS/400, iSeries, pSeries, xSeries, zSeries, z/OS, AFP,
Intelligent Miner, WebSphere, Netfinity, Tivoli, Informix,
i5/OS, POWER, POWER5, OpenPower and PowerPC are
trademarks or registered trademarks of IBM Corporation.
Adobe, the Adobe logo, Acrobat, PostScript, and Reader
are either trademarks or registered trademarks of Adobe
Systems Incorporated in the United States and/or other
countries.
Oracle is a registered trademark of Oracle Corporation.
UNIX, X/Open, OSF/1, and Motif are registered
trademarks of the Open Group.
Citrix, ICA, Program Neighborhood, MetaFrame,
WinFrame, VideoFrame, and MultiWin are trademarks or
registered trademarks of Citrix Systems, Inc.
HTML, XML, XHTML and W3C are trademarks or
registered trademarks of W3C®, World Wide Web
Consortium, Massachusetts Institute of Technology.
Java is a registered trademark of Sun Microsystems, Inc.
JavaScript is a registered trademark of Sun Microsystems,
Inc., used under license for technology invented and
implemented by Netscape.
MaxDB is a trademark of MySQL AB, Sweden.
SAP, R/3, mySAP, mySAP.com, xApps, xApp, SAP
NetWeaver, and other SAP products and services
mentioned herein as well as their respective logos are
trademarks or registered trademarks of SAP AG in
Germany and in several other countries all over the world.
All other product and service names mentioned are the
trademarks of their respective companies. Data contained
in this document serves informational purposes only.
National product specifications may vary.
These materials are subject to change without notice.
These materials are provided by SAP AG and its affiliated
companies ("SAP Group") for informational purposes only,
without representation or warranty of any kind, and SAP
Group shall not be liable for errors or omissions with
respect to the materials. The only warranties for SAP
Group products and services are those that are set forth in
the express warranty statements accompanying such
products and services, if any. Nothing herein should be
construed as constituting an additional warranty.
These materials are provided “as is” without a warranty of
any kind, either express or implied, including but not
limited to, the implied warranties of merchantability,
fitness for a particular purpose, or non-infringement.
SAP shall not be liable for damages of any kind including
without limitation direct, special, indirect, or consequential
damages that may result from the use of these materials.
SAP does not warrant the accuracy or completeness of the
information, text, graphics, links or other items contained
within these materials. SAP has no control over the
information that you may access through the use of hot
links contained in these materials and does not endorse
your use of third party web pages nor provide any warranty
whatsoever relating to third party web pages.
SAP NetWeaver “How-to” Guides are intended to simplify
the product implementation. While specific product
features and procedures typically are explained in a
practical business context, it is not implied that those
features and procedures are the only approach in solving a
specific business problem using SAP NetWeaver. Should
you wish to receive additional information, clarification or
support, please refer to SAP Consulting.
Any software coding and/or code lines / strings (“Code”)
included in this documentation are only examples and are
not intended to be used in a productive system
environment. The Code is only intended better explain and
visualize the syntax and phrasing rules of certain coding.
SAP does not warrant the correctness and completeness of
the Code given herein, and SAP shall not be liable for
errors or damages caused by the usage of the Code, except
if such damages were caused by SAP intentionally or
grossly negligent.
Disclaimer
Some components of this product are based on Java™. Any
code change in these components may cause unpredictable
and severe malfunctions and is therefore expressively
prohibited, as is any decompilation of these components.
Any Java™ Source Code delivered with this product is only
to be used by SAP’s Support Services and may not be
modified or altered in any way.