Troubleshooting Cisco Unified Contact Center Express PDF

Troubleshooting
Cisco Unified Contact

Center Express
BRKCCT-2015
Michael Green
Session Overview
 Explore some of the key changes with

Contact Center Express Version 8.0 and 8.5
 Demonstrate how to investigate problems

using available tools and logs
 Share troubleshooting and serviceability tips

and best practices
BRKCCT-2015 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 2
Agenda
 Contact Center Express Overview
 Troubleshooting Methodology
 Troubleshooting Contact Center Express Components

 Configuration, Database, High Availability, Log Analysis
 Contact Center Express Serviceability
 Q and A
What You Should Know
 Understanding of basic installation, configuration
and administration of Unified Contact Center
Express (CCX) solution
 Scope:
 Unified CCX 8.5(1) (mostly applies to 8.0(X) also)
 Two-Server High Availability (HA) over WAN deployment

model integrated with Cisco Unified Communications
Manager (CM) version 8.5(1)
If You Want to Play …
… You Need to Know the Basics
Source: ―Braid‖ by Jonathon Blow 2008

Contact Center Express Overview
Solution Components:
 Voice Gateway (VG)
 SIP, MGCP or H.323 voice gateway
V
 Allows incoming/outgoing calls from public switched
telephony network (PSTN)
 Communications Manager (CM)

 Responsible for call control and agent users
 Contact Center Express (CCX)

 Workflows play prompts and collect digits from
callers
 Queues calls for delivery to available agents
 Cisco Agent Desktop (CAD)

 IP Phones and PC‘s running CAD client
Solution Component Interactions
CCX Cluster Voice Gateway

Primary Cluster View Voice Gateway
Daemon V
Engine
Datastores
Secondary CAD
CM Cluster
Administration
UC Manager
CTI Manager
Agent
IP Phone
CAD Client
Unified CCX Components
Unified CCX
Engine CAD
Unified CM Telephony eMail LDAP Monitor Rec. and Statistics
CMT HTTP LRM VoIP Monitor
MRCP ASR/TTS VoIP Monitor Call/Chat Recording
RMCM Outbound Sync Browser and IPPA
Database RTR Enterprise Agent E-Mail
Database Agent Datastore Cluster View Daemon

Config Datastore
Application Administration
Historical Datastore
Repository Datastore Serviceability Administration
Deployment Models
Single-Server Non-HA Two-Server HA Single-Server Non-HA
CM Integration CM Integration CME Integration
V V
Voice Voice
PSTN Gateway PSTN Gateway PSTN
Voice Gateway
with CME
Single CM Cluster Two-Server CM Cluster Single

CCX Server CCX Cluster CCX Server
Agents
Agents Agents
Deployment Models — Two-Server HA with CM Integration
High Availability over LAN (HAoLAN) High Availability over WAN (HAoWAN)
V V
PSTN CM PSTN CM
VG VG
Two-Server Cluster
CCX Cluster
Agents WAN
V
Voice
Gateway CM Cluster Primary Secondary
Agents CCX Server CCX Server Agents
Summary:
 Solution Components and interactions
 VG, CM/CME, CCX, CAD agents
 Unified CCX Components

 CVD, Engine, CAD, Database, Administration
 Deployment Models
 Single-Server Non-HA with CM, Two-Server HA with CM
(HAoLAN and HAoWAN) and Single-Server Non-HA with
CME
You Can‟t just Go Marching into Battle Without a Plan…
… You Need a Strategy

Source: ―StarCraft 2‖ by Blizzard Entertainment 2010
Troubleshooting Methodology
Problem Description
IT DOESN‟T WORK!
“Oh…, you mean this morning about

half of the calls sent to „CTI Route
Point 2100‟ just sit in the queue and
are never delivered to any available
agent. This was working yesterday
before the „SalesEnquiry‟ script was
updated last night at about 9pm?”
 Was this working before? If yes, what changed?

 What is the frequency of the problem?
 What is the extent of the problem?
Steps to Solution
Think
Problem beyond
The fix
Solution
Summary:
 Clarify Problem Description

 Know what you are troubleshooting
 Take a logical and methodical approach

 Trial and error will only get you so far
Fine Tuning Your Car Can Be Critical When Racing …
… but You Need to Understand the Impact of

Configuration Changes
Source: ―Forza 2‖ by Microsoft 2007
Troubleshooting Contact Center
Express Components:
Configuration
Configuration
Configuration Mapping of Call Flow
Understand the impact of configurations in your call flow
Telephony
Provider
CTI Route Telephony Call Control

CTI Ports
Point Trigger Group
Application
RmCm
Provider
Agent
Device Script
Resource
Groups
Agent
Resources CSQ
User
Skills
Configuration
 Telephony Configuration and Synchronization
 Some configuration objects in CCX have corresponding
configurations in CM and need to be kept in sync
 This is done using the AXL interface on the CM
Unified CCX Server Unified CM Server
AXL AXL Web

CCX Admin
Provider Service
Telephony Provider JTAPI User
RmCm Provider RmCm User
Telephony Triggers CTI Route Points
Call Control Group CTI Ports
Configuration
 Changes to Telephony Synchronization in 8.5(1)
 Select which Telephony configurations
to check or sync:
 Improved Data Check Results show you exactly

which device or attribute is out of sync
Configuration
 Troubleshooting Telephony Config/Sync Issues
 While trying to create a new Telephony Trigger:
 Enable debug in MADM: ADM_CFG, LIB_AXL

Admin is initiating creating the new Trigger
%MADM-ADM_CFG-7-UNK:JTAPITriggerServlet.updateNewTrigger()-
Creating a new Trigger :2002
Admin intends to create a new RP and Line on CM

%MADM-ADM_CFG-7-UNK:JTAPITriggerUtil.createRPAndLineOnCCM() - routePoint=2002
%MADM-ADM_CFG-7-UNK:JTAPITriggerUtil.createRPAndLineOnCCM() - deviceName=Support
%MADM-ADM_CFG-7-UNK:JTAPITriggerUtil.createRPAndLineOnCCM() - Creating a LINE
Admin intends to create a new RP and Line on CM

%MADM-LIB_AXL-7-UNK:AXL-ExecutionCmd-160.CCMLineSOAPAdmin: try makeRequest()
on AXL: 192.168.12.1, AXLUser: admin, AXLPassword: XXXXXX
Configuration
Troubleshooting Telephony Config/Sync Issues
Admin sends XML SOAP request to CM AXL to recreate the new Line
Admin sends the AXL SOAP XML Request <addLine>
%MADM-LIB_AXL-7-UNK:AXL-ExecutionCmd-160.CCMLineSOAPAdmin: makeRequest() - Start REQUEST ====================
<newLine>
%MADM-LIB_AXL-7-UNK:POST /axl/ HTTP/1.1
Connection: keep-alive
<pattern>2002</pattern>
Host: 192.168.12.1:8443 <description>CRS Line description</description>
Authorization: Basic YWRtaW46Y2NidWRzMSE= ...
SOAPAction: "CUCM:DB ver=7.0" </newLine>
Accept: text/* Just look at </addLine>
Content-type: text/xml; charset="utf-8"
the XML in the
Cache-Control: no-cache
Pragma: no-cache SOAP body!
Content-length: 440 CM AXL is responding with failure response
<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"><SOAP-
<SOAP-ENV:Fault>
ENV:Header><seq>MADM_160</seq></SOAP-ENV:Header><SOAP-
<faultcode>SOAP-ENV:Client</faultcode>
ENV:Body><addLine><newLine><pattern>2002</pattern><description>CRS Line
<faultstring>Cannot
description</description><usage>CallPark</usage><routePartition insert oruuid=""/><callPickupGroup
uuid=""/><aarNeighborhood update pattern. A DN exists with the same pattern and
partition.</faultstring>
uuid=""/><shareLineAppearanceCSS></shareLineAppearanceCSS></newLine></addLine></SOAP-ENV:Body></SOAP-
ENV:Envelope> <detail>
350055: May 05 06:26:48.220 BST %MADM-LIB_AXL-7-UNK:AXL-ExecutionCmd-160.CCMLineSOAPAdmin: makeRequest() - End
REQUEST ==================
...
</detail>
</SOAP-ENV:Fault>
Investigating the AXL
messages between CCX and Admin fails to create the Line and Route Point for the new Trigger
%MADM-ADM_CFG-7-UNK:JTAPITriggerUtil.createRPAndLineOnCCM() –
CM can often provide detail on ERROR - Line couldn't be created/updated.
Telephony Configuration %MADM-ADM_CFG-7-UNK:JTAPITriggerServlet.createRPAndLineOnCCM() –
Created a Route Point = null
issues %MADM-ADM_CFG-7-UNK:JTAPITriggerServlet.createTriggerData() –
Error: Failed to create Route Point
Configuration
Cleaning Telephony Data Mismatches
 Prior to 8.5 Telephony Data configuration issues can
cause the configuration to get messy
 You can use the below steps to recreate the Telephony Data
configuration if encountering major issues with Data Sync
Recommended Steps to Recreate Telephony Triggers and Call Control Groups
1. From CCX Admin make a record of Trigger and Call Control Group
configurations
2. Delete Triggers and Call Control Groups
3. From CM Admin delete associated CTI Route Points and CTI Ports
• From Devices-> CTI Route Point search on Description with the Device Name
from the Trigger configuration to find and delete CTI Route Points
• From Devices-> Phones search on Description using the Device Name Prefix
from the Call Control Group configuration to find and delete all CTI Ports
4. From CM Admin delete relevant Unassigned DN’s from Route Plan Report
• Delete all the Unassigned DN‘s previously used by Route Point or Ports
5. Using CM Admin Route Plan Report double-check there are no DNs on the
ranges which CTI Route Point and CTI Ports will use
6. Recreate the Triggers and Call Control Groups from CCX Admin
Configuration
Telephony Subsystem Is Partial or Out Of Service
 Can be caused by Telephony Data mismatch, but

could also be unrelated
 Investigate MIVR logs to get better idea of cause
 Search backwards through logs for PARTIAL_SERVICE
or OUT_OF_SERVICE to find interesting part of the logs
Route Point and CTI Port addresses are out of service causing Telephony (JTAPI) subsystem to go out of service
...
%MIVR-SS_TEL-6-CTI_ADDRESS_EVENT:Receives: Event=CiscoAddrOutOfService,cause=1007,Addr=RP[num=2001]
%MIVR-SS_TEL-7-UNK:Route.setServiceState() from In Service to Out of Service for RP[num=2001]
...
%MIVR-SS_TEL-1-SS_OUT_OF_SERVICE:JTAPI subsystem in out of service: Failure reason=A number of route points are OOS –
RP[num=2001]; A number of CTI ports are OOS - TPG[id=0,state=PARTIAL_SERVICE]:
Ports[2508, 2507, 2506, 2505, 2500, 2504, 2503, 2502, 2501]
 Determine whether configuration mismatch is likely, or

more investigation between JTAPI client logs and CM CTI
Manager is required
Configuration
Agent User / Resource Synchronization
 Engine checks which CM End Users are configured
as Resources via AXL interface and syncs those
 Engine syncs Resources every 10 minutes
 Access CCX Administration Resource page to force sync
 Desktop Sync Service receives config update event

 Updates Resources/Agents in Desktop LDAP
configuration database
 Force a manual sync from Desktop Administrator
Unified CCX Server Unified CM Server
Config AXL Web

CCX Engine
Datastore Service
Sync End
Service Users
Configuration
Resource Synchronization Analysis
 Within 10 minutes of adding new agent ―newagent‖:
Resource Manager Subsystem initiates resource synchronization
%MIVR-SS_RM-7-UNK:--Last sync done at Fri May 06 02:23:02 BST 2011
%MIVR-SS_RM-7-UNK:the master-secondary status of this Engine is = true
%MIVR-SS_RM-7-UNK:--# Start Sync data inside RsrcMgr.configReaderSyncer
...
%MIVR-LIB_CFG-7-UNK:DBIAQConfiguration.dbSyncResources()--calling synchAgentData
%MIVR-LIB_CFG-7-UNK:DBIAQConfiguration.synchAgentData() - Begin read from CM DB
%MIVR-LIB_CFG-7-UNK:getAllAgents q BE + all agents having IPCC extension!!!1304645582597
Query for End Users which have “IPCC Extension” configured (End Users which are agents) using AXL
<executeSQLQuery>
<sql>SELECT FIRST 500 eu.userid, eu.pkid, np.dnorpattern, eu.firstname, eu.lastname, eu.middlename, eu.nickname FROM enduser AS
eu INNER JOIN endusernumplanmap AS enm ON eu.pkid = enm.fkenduser INNER JOIN numplan AS np ON enm.fknumplan = np.pkid
WHERE eu.tkuserprofile = 1 AND eu.status = 1 AND enm.tkdnusage = 2 ORDER BY eu.pkid
</sql>
</executeSQLQuery>
AXL response with the rows which are the results returned by the SQL query
<axl:executeSQLQueryResponse xmlns:axl="http://www.cisco.com/AXL/API/7.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<return>
<row><userid>uccxlondonagent</userid> ... <dnorpattern>2302</dnorpattern> ... </row>
<row><userid>uccxsuper</userid> ... <dnorpattern>2301</dnorpattern> ... </row>
<row><userid>newagent</userid> ... <dnorpattern>2103</dnorpattern> ... </row>
<row><userid>uccxparisagent</userid> ... <dnorpattern>2401</dnorpattern> ... </row>
</return></axl:executeSQLQueryResponse>
...
%MIVR-LIB_CFG-7-UNK:DBIAQConfiguration.synchAgentData() - retrieved agents are: 4
Configuration
Resource Synchronization Analysis
Query for End Users which are configured as Supervisors using AXL
<executeSQLQuery>
<sql>SELECT FIRST 500 eu.userid, eu.pkid, cs.issupervisor FROM enduser AS eu INNER JOIN crsuserclusterspecific AS cs ON
cs.fkenduser = eu.pkid AND cs.clustername = '1302270911000' WHERE eu.tkuserprofile = 1 AND eu.status = 1 AND cs.issupervisor = 't‗
ORDER BY eu.pkid</sql>
</executeSQLQuery>
AXL response with the rows which are the results returned by the SQL query
<axl:executeSQLQueryResponse xmlns:axl="http://www.cisco.com/AXL/API/7.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<return>
<row><userid>uccxsuper</userid> ... <issupervisor>t</issupervisor></row>
</return>
</axl:executeSQLQueryResponse>
...
%MIVR-LIB_CFG-7-UNK:DBIAQConfiguration.synchAgentData() - retrieved supervisorts are: 1
The query results are compared with the database, Resources are added, updated or marked as Inactive accordingly
%MIVR-LIB_CFG-7-UNK:DBIAQConfiguration.synchAgentData() - Begin read from CRS DB
...
%MIVR-LIB_CFG-7-UNK:DBIAQConfiguration.synchAgentData() - updated agent newagent
%MIVR-LIB_CFG-7-UNK:DBIAQConfiguration.synchAgentData() completed. Mark deleted:0, Updated:0, Added:1
This triggers CTI to sent update events which causes Sync Service to update agent changes in local LDAP configuration database
%MIVR-ICD_CTI-7-UNK:MsgHandler : Sent : { CONFIG_AGENT_EVENT
----------
DEBUG [0x34baba0] NewAcmiSync.cpp[271] processCTIMessage: Received CONFIG_AGENT_EVENT and SynAgents.
...
DEBUG [0x34baba0] AcmiSynchronizer.cpp[518] SynAgents: new agent ->-> name< newagent>
Configuration
Summary
 Configuration Mapping of Call Flow
 Telephony Configuration and Synchronization

 How it works, 8.5(1) Changes and troubleshooting
 Agent End User and Resource Synchronization
Playing Cooperatively …
… Where Two Is Better than One
Source: ―Lego Batman‖ by TT Games 2008

Express Components:
High Availability
High Availability
Unified CCX — A Fault Tolerant Solution
 Utilizes a Unified Communications network

 Composed of Cisco routers and switches to provide a highly
available data network with many options for redundancy
 Unified CM utilizes a clustered approach

 Devices (voice gateways, IP Phones, Route Points and CTI
Ports) can register with a primary, secondary or tertiary CM
 VGs at multiple locations using different service providers
 CCX communicates with primary and secondary CTI Manager
 Unified CCX two-server High Availability cluster

 Providing redundancy for CCX Engine, database, Monitoring
and Recording components
High Availability
CCX Component Redundancy
Monitoring
Primary CCX Server Secondary CCX Server • If desktop
Engine monitoring fails
M Engine Engine S
• First started SPAN port
becomes Master monitoring can be
Databases Databases
• Primary is used as backup
preferred Master Monitoring Monitoring • Phone
• Other node will configured with
be Standby Recording Recording SPAN port
monitoring tied to
one node
Databases
• Comprised of Historical, Agent, Repository and Recording
Config Datastores (HDS, ADS, RDS and CDS) • Recording tasks load balanced in
•DB mastership prefers Engine Master (unless round robin fashion
DB is down on Engine Master) • Recordings stored on physical server
• Data is read from / written to DB Master where recording task took place
• HDS/ADS/RDS has Publisher/Subscriber and • Recording components act as backup
syncs using Informix Enterprise Replication for each other
• CDS uses distributed transactions (both • Requires Monitoring component
databases must be operational for updates)
High Availability
Failover Detection Mechanism
 Cluster View Daemon (CVD) sends and monitors
heartbeats between CCX nodes
HA over LAN HA over WAN

Heartbeat sent every 500 ms Heartbeat sent every 1 second
Failover after 5 missed heartbeats Failover after 10 missed heartbeats
Failover occurs within 5 seconds Failover occurs within 12 seconds
Primary CCX Server Secondary CCX Server
Port
UDP Heartbeats 1996
CVD CVD
Port
1996 UDP Heartbeats
High Availability
LAN versus WAN Deployment
 LAN: Both nodes share AXL, Unified CM Telephony
and RMCM Provider list configuration
High Availability over LAN (HAoLAN) High Availability over WAN (HAoWAN)
CM Publisher CM Subscriber CM Publisher CM Subscriber
WAN
Primary CCX Secondary CCX Primary CCX Secondary CCX
 WAN: Both nodes have unique AXL, Unified CM

Telephony and RMCM Provider list configuration
High Availability
Example Scenario: Engine Failure
Primary Engine Fails and is detected by Primary CVD

Primary CCX Server Secondary CCX Server %MCVD-NODE_MGR-0-NODE_MGR_SERVICE_CRASH_DETECTED:
Node manager detected service crash: service name=Cisco Unified CCX Engine
CVD CVD
M ? S S ? M
%MCVD-CVD-7-UNK:Drop master for service Cisco Unified CCX Engine,
because newState=SHUTDOWN
! %MCVD-NODE_MGR-7-UNK:Service:Cisco Unified CCX Engine failed due to reason:
Engine Engine Crashed
Databases Databases Master Election initiated on Primary CVD

%MCVD-CLUSTER_MGR-7-UNK:Post Convergence Event:
CONVERGENCE_STARTED, name=Cisco Unified CCX Engine
Monitoring Monitoring %MCVD-CLUSTER_MGR-7-UNK:JavaService66: Cisco Unified CCX Engine on
node 1 change master from true to false
%MCVD-CLUSTER_MGR-7-UNK:Post Master Event: MASTER_DROPPED,
Recording Recording name=Cisco Unified CCX Engine, node=1
Master Election initiated on Secondary CVD

 Engine Failure detected CONVERGENCE_STARTED, name=Cisco Unified CCX Engine

%MCVD-CLUSTER_MGR-7-UNK:JavaService66: Cisco Unified CCX Engine on
 Master Election initiated name=Cisco Unified CCX Engine, node=1
Secondary CVD elects Secondary as Master

 Master Election completed %MCVD-CLUSTER_MGR-7-UNK:JavaService167: Cisco Unified CCX Engine on
node 2 change master from false to true
%MCVD-CLUSTER_MGR-7-UNK:Post Master Event: MASTER_ELECTED,
name=Cisco Unified CCX Engine, node=2
 Failover completed
CONVERGENCE_COMPLETED, name=Cisco Unified CCX Engine
High Availability
Example Scenario: Lost Connectivity
Secondary CVD detects missing keep-alives from Primary CVD

Primary CCX Server Secondary CCX Server %MCVD-CVD-5-HEARTBEAT_MISSING_HEARTBEAT:
CVD does not receive heartbeat from node for a long period:
CVD CVD
M ? ? S ? M
nodeId=1,dt=2049
.....
%MCVD-CVD-5-HEARTBEAT_MISSING_HEARTBEAT:
Engine ! Engine CVD does not receive heartbeat from node for a long period: nodeId=1,dt=10245
%MCVD-CVD-4-HEARTBEAT_SUSPECT_NODE_CRASH:CVD suspects node crash:
state=Heartbeat State,nodeInfo=Node id=1 ip=192.168.12.5
Databases Databases state=Convergence State,nodeInfo=Node id=1 ip=192.168.12.5
Monitoring Monitoring Master Election initiated on Secondary CVD

Recording Recording 7631: Apr 28 07:45:13.986 CEST %MCVD-CLUSTER_MGR-7-UNK:JavaService66:
Cisco Unified CCX Engine on node 1 change master from true to false
7632: Apr 28 07:45:13.986 CEST %MCVD-CLUSTER_MGR-7-UNK:Post Master
Event: MASTER_DROPPED, name=Cisco Unified CCX Engine, node=1
 Primary lost connectivity Secondary CVD elects Secondary as Master

node 2 change master from false to true
 Master Election initiated

 Master Election completed

High Availability
Example Scenario: Partitioned Network
Primary CCX Server Secondary CCX Server Primary CVD detects missing Heartbeats
and assumes Primary has failed
CVD CVD %MCVD-CVD-5-HEARTBEAT_MISSING_HEARTBEAT:
M M S ? M CVD does not receive heartbeat from node for a long period: nodeId=2,dt=10197
state=Convergence State,nodeInfo=Node id=2 ip=192.168.13.5
Engine Engine
!
Databases Databases Secondary CVD detects missing Heartbeats
and assumes Primary has failed
%MCVD-CVD-5-HEARTBEAT_MISSING_HEARTBEAT:
Monitoring Monitoring CVD does not receive heartbeat from node for a long period: nodeId=1,dt=10242
state=Heartbeat State,nodeInfo=Node id=1 ip=192.168.12.5
Recording Recording
Secondary CVD performs Master Election
 WAN network failure

%MCVD-CLUSTER_MGR-7-UNK:JavaService66:
Cisco Unified CCX Engine on node 1 change master from true to false
%MCVD-CLUSTER_MGR-7-UNK:Post Master
Event: MASTER_DROPPED, name=Cisco Unified CCX Engine, node=1
 Missing Heartbeats detected

Secondary CVD elects Secondary as Master
 Master Election performed node 2 change master from false to true
 Failover completed CONVERGENCE_COMPLETED, name=Cisco Unified CCX Engine
High Availability
Example Scenario: Recovery from Partitioned Network
Primary CCX Server Secondary CCX Server Primary CVD detects Secondary and keeps Master
%MCVD-CVD-7-UNK:Split after network partition is detected, new nodeId=2
%MCVD-CVD-7-UNK:Engine bestCandidate runs on nodeId=1
CVD CVD
M ? S
because primaryEngineComputerName=UC85CCXPRI
M M %MCVD-CVD-7-UNK:Master Cisco Unified CCX Engine conditional-Keep-LocalMaster
from localServiceCisco Unified CCX Engine on node 1
Engine Engine
!
Databases Databases Secondary CVD detects Primary and drops Master
%MCVD-CVD-7-UNK:Split after network partition is detected, new nodeId=1
%MCVD-CVD-7-UNK:Engine bestCandidate runs on nodeId=1
because primaryEngineComputerName=UC85CCXPRI
Monitoring Monitoring %MCVD-CVD-7-UNK:Master Cisco Unified CCX Engine DropLocalMaster from
localServiceCisco Unified CCX Engine on node 2, conditional=true
Recording Recording
Secondary CVD performs Master election and drops Master
 WAN network restored %MCVD-CLUSTER_MGR-7-UNK:JavaService167: Cisco Unified CCX Engine on
 CVDs detect dual Masters
Secondary CVD completes Master election
 Master Election performed %MCVD-CLUSTER_MGR-7-UNK:Post Convergence Event:
High Availability
Summary
 CCX is a fault tolerant solution

 UC Network, Unified CM, Unified CCX
 Unified CCX redundancy

 Engine, Databases, Monitoring and Recording
components
 Failover detection mechanism
 HA differences between LAN and WAN
 Example Scenarios
 Engine failure, lost connectivity, partitioned network
Where Are We Going to Put All this Stuff?
… Better Make Sure We Don‟t Lose Anything!
Source: ―World of WarCraft‖ by Blizzard Entertainment 2004

Express Components:
Database
Database
Datastore—Database Mapping and Purpose
 Each logical Datastore has a different purpose and
maps to an Informix DB
Used to store all the data to Historical

generate historical and real-
time reports
Datastore
db_cra
Stores all the configuration Config
data from CCX Admin such
as Applications, CSQs, etc
Datastore
Stores scripts, prompt files, Repository

custom documents and
grammars
Datastore db_cra_repository
Used by the Desktop Agent

services to store recent call
data for agent statistics
Datastore
fcrassvr
Database
Informix Database Enterprise Replication
 A replicate transaction is performed, copied from
the log and grouped with others to be sent or stored
Source Server Transport Target Server
Application
Database Snoopy Database
Grouper Ack Queue Data Sync
Logical
Log
Send Network Receive
Spool
Queue Queue
Source: IBM Informix Enterprise Replication Guide

Database
CCX Tables DB Using Informix Enterprise Replication (ER)
db_cra db_cra_repository fcrassvr
Replicated Tables: Replicated Tables: Replicated Tables:

agentconnectiondetail documentsfiletbl fcrascalllogtoday
agentstatedetail documentsfoldertbl fcrascalllogweek
contactcalldetail grammarsfiletbl fcrasrecordlog
contactqueuedetail grammarsfoldertbl fcrasstatelogtoday
contactroutingdetail latestsynchedtime latestsynchedtime
eemactiveemail promptsfiletbl
eemcontactemaildetail promptsfoldertbl
eememailagentstatedetail scriptsfiletbl
eememailstatusdescription scriptsfoldertbl
eemqueueagentdetail sysgrammarsfiletbl
eemreasoncodedescription sysgrammarsfoldertbl
eemstatedescription
latestsynchedtime
monitoredresourcedetail
remotemonitoringdetail
ER
 Tables participating in ER are Source/Target Source/Target
replicated bi-directionally
Database
Config Datastore
 Does not use Enterprise Replication
 If both Publisher and Subscriber are available, the
update is made on both servers
 If one database is down—no change is made, this is to
preserve the integrity of the configuration database
Update with Subscriber
Publisher both DBs
db_cra
available
db_cra
succeeds Config Datastore Tables
Config Datastore Tables
P P areacode
campaign
resourcegroup
resourceskillmapping
areacode resourcegroup
campaigncsqmap rmoncsqconfig
campaign resourceskillmapping
campaigndata rmonresconfig
campaigncsqmap rmoncsqconfig
contactservicequeue rmonuser
campaigndata rmonresconfig
crsapplication skill
contactservicequeue
crsapplication
rmonuser
skill O crsgroup
crstrigger
skillgroup
supervisor
crsgroup skillgroup Update with dialinglist team
crstrigger supervisor
dialinglist team one DB profileidmapping teamcsqmapping
resource workflowtask
profileidmapping teamcsqmapping available
resource workflowtask
fails
Database
Make Config Changes with Secondary Database Down
 Both databases must be

available for changes to be
committed to Config Datastore
(CDS) CCX Administration (MADM) Logs:
%MADM-ADM_CFG-7-UNK:ICDServlet :: Exception occurred
 Modifying Skills, CSQ‘s, The SUBSCRIBER node of the HA is not available
%MADM-ADM_CFG-3-ADM_EXCEPTION:Unknown ADM Exception:
Exception=java.lang.RuntimeException: The SUBSCRIBER node of the HA is not available
Applications, etc
 Temporarily Disable CDS on

Subscriber Node
 CCX Serviceability Admin-> Tools-
> Datastores-> Replication
 Disable CDS and HDS
 Enable after Secondary restored
Database
Historical Datastore
 Engine Master writes Historical Reporting data
records to Database Master
 Data gets replicated to Standby server
 Historical Reporting queries run on Standby server
 Datastore Mastership follows Engine Mastership
Reporting
Primary Server Secondary Server queries
Records M S run on
written to Database
Database Standby
Master ER server
Historical Historical
Datastore Datastore
Data replicated
(1-3min delay)
Database
Historical Reporting—Missing Data
 Verify data is in the DB to narrow down the scope

CCX Primary: CCX Secondary:
admin:run uccx sql db_cra SELECT endDateTime, sessionID, admin:run uccx sql db_cra SELECT endDateTime, sessionID,
sessionSeqNum, nodeID FROM ContactCallDetail WHERE sessionSeqNum, nodeID FROM ContactCallDetail WHERE
endDateTime between '2011-04-23 08:00:00' and '2011-04- endDateTime between '2011-04-23 08:00:00' and '2011-04-
23 08:30:00' 23 08:30:00'
ENDDATETIME SESSIONID SESSIONSEQNUM NODEID ENDDATETIME SESSIONID SESSIONSEQNUM NODEID
------------------------------------------ ------------------------------------------
2011-04-23 08:03:50.144 11000002142 0 1
2011-04-23 08:12:25.46 11000002143 0 1 No records found Database records existing
2011-04-23 08:24:13.666 11000002144 0 1 in Primary/Master DB
2011-04-23 08:26:25.649 11000002145 0 1 Command successful.
admin: but missing in
Command successful. Secondary/Standby DB
admin:
 Reporting queries are run from Secondary DB

Database
Database Replication Status
 View from CCX Admin or CLI
 Connected / Active state shows healthy replication
 Dropped / Timed Out status indicates replication issue
Tools-> Datastore Control Center->
Replication Servers
admin:show uccx dbreplication servers

This operation may take a few minutes to complete. Please wait...
SERVER ID STATE STATUS QUEUE CONNECTION CHANGED

-----------------------------------------------------------------------
g_uc85ccxpri_uccx 1 Active Local 0
g_uc85ccxsec_uccx 2 Active Connected 0 May 9 18:07:55
CLI Command: Command successful.
:show uccx dbreplication servers admin:
Database
Resetting a Broken Replication
 Replication can be Removed if Subscriber is
unavailable for too long and send queues exceeded
 Typically 3-4 days (*can vary with load)
Alert Raised!
 Alert will be raised DBReplicationStopped
 Reset Replication from Tools-> Datastore Control

Center-> Replication Servers menu from CCX
Serviceability Admin
Issuing a Reset Replication causes the following to occur:
1. Remove database replication
:utils uccx dbreplication teardown
2. Setup database replication
:utils uccx dbreplication setup
3. Perform data repair process
:utils uccx dbreplication repair all
Database
Historical Reporting—Missing Data
 If no Database is available, historical data records
are written to flat file and alert is raised
Primary Server Replication Secondary Server
M ! won‘t be S !
occurring
DB DB Alert Raised!
HistoricalDataWrittenToFiles
If both DBs are

down Engine
File System File System
writes records
directly to file
 Fix Database issue then manually import

records from file system with Tools->
Historical Reporting-> File Restore menu
from CCX Admin
Database
Broken Replication—Further Analysis
 From Real-Time Monitoring Tool (RTMT)

collect logs from Cisco Unified CCX Database
 Investigate uccx_repl_output_util.log file for issues

related to database replication
Database
Summary
 Datastore—Database Mapping and Purpose
 How CCX uses Informix Database Enterprise
Replication
 Config Datastore
 How updates work, how updates can be made without
Subscriber
 Historical Datastore
 Where data is written to / read some and troubleshooting
scenario
 Database Replication Status Check and Reset
Slow Motion Action Replay …
… Stop Time and Investigate what Really Happened
Source: ―FIFA Soccer 11‖ by EA 2010

Express Components:
Log Analysis
Log Analysis
Where Do We Start When Faced with a New Problem?
 Log Analysis?
 No! Analyzing logs is just another Troubleshooting
Tool we can use as part of the investigation in our
Troubleshooting Methodology to approach problem
solving
 Troubleshooting Methodology—Clarify problem
description, determine possible causes, investigate
and test likely causes
 Rule out the Basics—Are required components in
service? Check relevant configuration and confirm
environmental factors such as network configuration
and hardware/software version compatibility
Log Analysis
How to Analyze Logs
 Practice—learn by doing
 Understand component interaction and lifecycle
 Compare failure scenarios with working scenarios
 Find interesting keywords such as EXCEPTION,
error, failure, timeout
Log Analysis
Agent Login—Lifecycle
CCX Cluster
4 6
2
1
3
CM Cluster Agent
1. Agent sends Login request to CCX

2. CCX sends authentication request to CM via AXL
Call Control 3. CM authenticates the ID/password against LDAP
and CTI data 4. CM responds to the AXL request with success
5. CCX verifies extension is controlled by RM user
6. CCX notifies agent of successful Login
Log Analysis
Agent Login—Log Analysis
CAD sends login request (set agent state request)
DEBUG [0x834] PhoneDev.cpp[500] SetAgentState: SetAgentState Begin: Set Agent State AS_LOGIN , Invoked ID 3, AgentID uccxlondonagent,
DeviceID 2302, PeripheralID 1, reasonCode 1, ForcedFlag 0
CCX receives the set agent state request message

%MIVR-SS_RM-7-UNK:Processing msg: CTISetAgentStateReqMsg (Rsrc:uccxlondonagent InvokeID:3 State:LOGIN Forced:False)
CCX sends AXL request to CM to get the properties of the agent

<executeSQLQuery><sql>
SELECT eu.pkid, eu.firstname, eu.lastname, ... FROM enduser ... WHERE eu.tkuserprofile = 1 AND eu.status = 1 AND eu.userid = 'uccxlondonagent‗
</sql></executeSQLQuery>
CCX gets AXL response with agent properties
<return><row>
<pkid>386a2b6a-95fb-3fea-3621-8434d12dcc25</pkid><firstname/><lastname>UCCX London Agent</lastname> ...
</row></return>
CCX authenticates agent via AXL request

%MIVR-SS_RM-7-UNK:Trying to authenticate agent uccxlondonagent
<doAuthenticateUser><userid>uccxlondonagent</userid><password>XXXXXX</password></doAuthenticateUser>
CCX gets AXL response with acknowledgement agent is successfully authenticated
<return><userAuthenticated>true</userAuthenticated><code>0</code></return>
%MIVR-SS_RM-7-UNK:Successfully authenticated agent uccxlondonagent at extension 2302
CCX verifies agent extension is monitored by RM user and adds a mapping to it

%MIVR-SS_RM-7-UNK: The mapping 2302-uccxlondonagent has been added
Agent uccxlondonagent is successfully transitioned to UNAVAILABLE (Not Ready) state
%MIVR-SS_RM-7-UNK:Rsrc: uccxlondonagent New State:UNAVAILABLE Old State:LOGOFF Reason code:32760
CAD receives confirmation agent is successfully logged in

DEBUG [0x834] PhoneDev.cpp[569] SetAgentState: SetAgentState End: Set Agent state to AS_LOGIN successful
Log Analysis
Agent Login—What Can Go Wrong?
 Incorrect Agent ID or Password

 Typically end user error indicates this
 CM configuration should be verified or AXL interactions in
MIVR logs can be further investigated
 Agent Extension mapping fails

 Verify extension associated with RM User and device has
agent supported configuration
 Investigate CCX JTAPI client / CM CTI Manager logs
 Login is timing out

 Common cause is authentication takes too long with CM
Active Directory integration for LDAP
 Ensure critical services are all in full service
Log Analysis
Call Arrival—Lifecycle
CCX Cluster
PSTN Caller
3 4
1 5
Voice Gateway 2 CM Cluster
PSTN Call 1&2. VG route request sent to CM CTI Route Point

3. JTAPI route request sent to CCX
IP Voice 4. CCX requests CM redirect call to CTI Port
5. CM offers call to CCX CTI Port (ringing)
Call Control
and CTI data
Log Analysis
Call Arrival—Finding a Call in the MIVR Logs
 Note details to uniquely identify the call

 Time of call, calling number, called number, duration, etc
 Goal is to find Call ID so we can filter logs with it

 Search on Call.Received to see all call arrivals
 Use regular expression Call.Received().*<CallingNum>
along with checking timestamp to find unique call
Search String:
Call\.Received.*2201
Search Results:
760983: May 19 08:31:15.859 BST %MIVR-SS_TEL-7-UNK:Call.received()
JTAPICallContact[id=0,implId=1007/2,state=STATE_RECEIVED_IDX,inbound=true,App name=Sales,task=null,
session=null,seq num=1,cn=2001,dn=2001,cgn=2201,ani=null,dnis=null,clid=null,atype=DIRECT,lrd=null,ocn=2001,
route=RP[num=2001],OrigProtocolCallRef=00000000000003EF02207EDF00000000,DestProtocolCallRef=null,TP=null
Call ID:
1007/2
Log Analysis
Call Arrival—Log Analysis
CCX receives new call 1007/2 on CTI Route Point 2001 from DN 2201 invoking application Sales
%MIVR-SS_TEL-7-UNK:Route Connection=[2001:pt_UCCX:1/(P1-jtapi_1) GCID=(2,1007)->ACTIVE]->OFFERED, reason=1, Event= CallCtlConnOfferedEv,
cause=100, metacode=129, isMaster=true
%MIVR-SS_TEL-7-UNK:Call.received() JTAPICallContact[id=0,implId=1007/2,state=STATE_RECEIVED_IDX,inbound=true,App name=Sales,task=null,
session=null,seq num=-1,cn=2001,dn=2001,cgn=2201,ani=null,dnis=null,clid=null,atype=DIRECT,lrd=null,ocn=2001,route=RP[num=2001],...
CCX associates call with available CTI Port (2508) and requests to redirect the call from RP 2001 to Port 2508
%MIVR-SS_TEL-7-UNK:Route Connection: [2001:pt_UCCX:1/(P1-jtapi_1) GCID=(2,1007)->ACTIVE]->OFFERED, CTI Port selected:
TP[id=0,implId=2508,state=IN_USE]
%MIVR-SS_TEL-7-UNK:Call.associated() JTAPICallContact[id=0,implId=1007/2,state=STATE_RECEIVED_IDX,inbound=true,App name=Sales,task=null,
session=21000000001,seq num=0,cn=2001,dn=2001,cgn=2201,ani=null,dnis=null,clid=null,atype=DIRECT,lrd=null,ocn=2001,route=RP[num=2001],
OrigProtocolCallRef=00000000000003EF02207EDF00000000,DestProtocolCallRef=null,TP=2508]
%MIVR-SS_TEL-7-UNK:Route Connection: [2001:pt_UCCX:1/(P1-jtapi_1) GCID=(2,1007)->ACTIVE]->OFFERED has 1 current sessions active.
Telephony (JTAPI) call control events causing call redirect to occur, call will be ringing (alerting) on CTI Port
%MIVR-SS_TEL-7-UNK: Got CallActiveEv, ConnCreatedEv, ConnInProgressEv, CallCtlConnOfferedEv, ConnCreatedEv, ConnConnectedEv,
RING RING... CallCtlConnEstablishedEv, ConnCreatedEv, events on the AddressCallObserver.
RING RING... %MIVR-SS_TEL-7-UNK: Got ConnInProgressEv, CallCtlConnOfferedEv, events on the AddressCallObserver.
%MIVR-SS_TEL-7-UNK: Got ConnDisconnectedEv, CallCtlConnDisconnectedEv, events on the AddressCallObserver.
%MIVR-SS_TEL-7-UNK:Call.accepted() JTAPICallContact[id=0,implId=1007/2,state=STATE_ACCEPTED_IDX,inbound=true,App name=Sales,task=null,
OrigProtocolCallRef=00000000000003EF02207EE000000000,DestProtocolCallRef=null,TP=2508]
%MIVR-SS_TEL-7-UNK:CallID:0 MediaId:1007/2, TerminalConnection to Terminal: UCCX_2508 is RINGING,
[2508:pt_UCCX:1/(P1-jtapi_1) GCID=(2,1007)->ACTIVE]->ALERTING
The Contact is attributed to a Task ID (workflow/script)

%MIVR-SS_TEL-7-UNK:Call.attributed() JTAPICallContact[id=0,implId=1007/2,state=STATE_ATTRIBUTED_IDX,inbound=true,App name=Sales,task=null,
OrigProtocolCallRef=00000000000003EF02207EE000000000,DestProtocolCallRef=null,TP=2508]
%MIVR-SS_TEL-7-UNK:CallID:0 MediaId:1007/2 Task:28000000002 associated with Task ID: 28000000002
Log Analysis
Call Arrival—What Can Go Wrong?
BEEP...
BEEP...
 Immediate busy tone BEEP...
 Typically a call control routing issue on CM: Verify Calling

Search Space and Partitions, check for overlapping
patterns, verify Locations and Regions (bandwith/codec)
 If call cannot be found in MIVR logs then it is not even
being routed to CCX from CM, investigate at CM or VG
 4 seconds of silence followed by busy tone

 CTI Accept Timer expires if CM does not receive redirect
request in response to route request after 4 seconds
 Typically no CTI Ports are available due to call volume,
Telephony Subsystem or CTI Port registration issues
1... 2... 3... 4...
BEEP...
BEEP...
Log Analysis
Call IVR and Queue Treatment—Lifecycle
2
CCX Cluster
CCX
PSTN Caller 3 Script
1 6
4
Voice Gateway 5 CM Cluster
PSTN Call 1. Call is ringing at CTI Port

2. CCX invokes Workflow Script
IP Voice 3. Accept Step directs CCX to answer call on CTI Port
4. Typically prompts are played
Call Control 5. and digits may be captured
and CTI data 6. Call queued to Contact Service Queue (CSQ)
Log Analysis
Call IVR and Queue Treatment—Log Analysis
CCX receives new call 1007/2 on CTI Route Point 2001 from DN 2201 invoking application Sales
%MIVR-ENG-7-UNK:Execute Task 28000000002
Application: App[name=Sales,type=Cisco Script Application,id=0, ... ,script=SSCRIPT[icd.aef], ...
Trigger: ContactApplicationTrigger ... Name=2001 ...
CCX Script Accept Step directs CTI Port to Answer the Call
%MIVR-ENG-7-UNK:Execute step of Task 28000000002 : Accept (--Triggering Contact--)
CTI and Telephony (JTAPI) events showing the call is answered

%MIVR-ICD_CTI-7-UNK:EventHandler: posting {BEGIN_CALL_EVENT: Socket:Socket: null monitoredDeviceDN:null, connectionCallID: 33555439, ...
%MIVR-ICD_CTI-7-UNK:EventHandler: posting {CALL_DELIVERED_EVENT: Socket:Socket: null monitoredDeviceDN:2508, connectionCallID: 33555439, ...
%MIVR-SS_TEL-7-UNK:Call.answered() JTAPICallContact[id=0,implId=1007/2,state=STATE_ANSWERED_IDX,inbound=true, ...
%MIVR-ICD_CTI-7-UNK:EventHandler: posting {CALL_ESTABLISHED_EVENT: Socket:Socket: null monitoredDeviceDN:2508, connectionCallID: 33555439,
%MIVR-SS_TEL-7-UNK:CallID:0 MediaId:1007/2 Task:28000000002 com.cisco.jtapi.CiscoMediaOpenLogicalChannelEvImpl received
%MIVR-SS_TEL-7-UNK:CallID:0 MediaId:1007/2 Task:28000000002 com.cisco.jtapi.CiscoRTPInputStartedEvImpl received
%MIVR-SS_TEL-7-UNK:CallID:0 MediaId:1007/2 Task:28000000002 com.cisco.jtapi.CiscoRTPOutputStartedEvImpl received
Thank you Prompt is played to call and attempts to allocate resource from CSQ Sales (“ESD 1”)
for calling... %MIVR-ENG-7-UNK:Execute step of Task 28000000002 : Play Prompt (--Triggering Contact--, WelcomePrompt)
%MIVR-ENG-7-UNK:Execute step of Task 28000000002 : Select Resource (--Triggering Contact-- from CSQ)
%MIVR-SS_RM-7-UNK:ESD 1 Sales .allocateRsrc(33555439 [1007/2])
No resources (agents) are available in the CSQ so call is played some prompts while queued
%MIVR-SS_RM-7-UNK:Cannot allocate an available resource b/c none is present in Sales
%MIVR-ENG-7-UNK:Execute step of Task 28000000002 : queueLoop:
%MIVR-ENG-7-UNK:Execute step of Task 28000000002 : Play Prompt (--Triggering Contact--, QueuePrompt)
... All of our representatives are assisting other

callers at this time. Your call is very important
to us, please stay on the line and your call will
handled in the order it was received ...
Log Analysis
Call IVR and Queue Treatment—What Can Go Wrong?
 Caller hears Error Prompt I’m sorry, we

are currently
 Typically a straight-forward scripting or configuration experiencing

system
problems...
issue easily diagnosed with log analysis: (1) Search for
last instance of UNRECOV_ERROR and (2) find
triggering exception prior to default error handler
%MIVR-ENG-7-UNK:Execute step of Task 18000000001 : Select Resource (--Triggering Contact-- from CSQ)
%MIVR-APP_MGR-3-TASK_ABORTED:Application task aborted: Application=App[name=Sales,...
%MIVR-APP_MGR-3-EXCEPTION:com.cisco.wfapi.WFExecutionException: Failed to load CSQ because CSQ: 5ales is invalid
...
%MIVR-SS_TEL-7-UNK:Call.transferring(UNRECOV_ERROR) JTAPICallContact[id=0,implId=185/1,state=STATE_ANSWERED_IDX,inbound=true,
App name=Sales,task=18000000001,session=6000000001,seq num=0,cn=3000,dn=3000,cgn=1902,ani=null,dnis=null,clid=null,atype=DIRECT,
lrd=null,ocn=3000,route=RP[num=3000],TP=3100]
 Unexpected behavior
 Step through processing of script steps to narrow down
issue or investigate any script exceptions
Log Analysis
Call Delivery to Agent—Lifecycle
CCX Cluster
PSTN Caller
1 2
3
5
CM Cluster 4 Agent
Voice Gateway
PSTN Call 1. Agent goes into Ready state

2. CCX assigns call and puts Agent in Reserved state
IP Voice 3. CCX requests to consult transfer call
4. Agent answers call and goes to Talking state
Call Control 5. Call ends and Agent placed back into Ready state
and CTI data
Log Analysis
Call Delivery to Agent—Log Analysis
CCX receives request from agent uccxlondonagent to transition into Available (Ready) state
%MIVR-ICD_CTI-7-UNK:ClientConnMgr: Processing msg on socket:Socket[addr=192.168.2.102,port=53461,localport=12028] Msg is {length=57
type=SET_AGENT_STATE_REQ,invokeId=22,agentState=AVAILABLE,eventReasonCode=0,forcedFlag=False,agentID=uccxlondonagent,
agentDN=2302,agentPwd=***** }
%MIVR-SS_RM-7-UNK:Rsrc: uccxlondonagent New State:AVAILABLE Old State:UNAVAILABLE Reason code:0
CCX assigns available agent (uccxlondonagent) to queued call (1007/2) and changes agent to Reserved state
%MIVR-SS_RM-7-UNK:ESD 1 .getHighestPriorityContact() returning 33555439 [1007/2]
%MIVR-SS_CM-7-UNK:RmCm contact 33555439[1007/2] (0) .setAllocatedResource(uccxlondonagent)
%MIVR-SS_RM-7-UNK:Rsrc: uccxlondonagent New State:RESERVED Old State:AVAILABLE Reason code:0
%MIVR-ICD_CTI-7-UNK:EventHandler: posting {AGENT_STATE_EVENT: Socket:Socket: null monitoredDeviceDN:2302, agentDN:2302,
agentID:uccxlondonagent, monitorID = 0, stateDuration = 0, agentstate = RESERVED, eventreasoncode = 0, agentID = uccxlondonagent,
agentExtension = 2302, agentID_Long = uccxlondonagent } to outboundQ
CCX interrupts the workflow script to prepare to transfer the call to the agent
%MIVR-SS_RM-7-UNK:Task: 28000000002 Start interrupting the task.
%MIVR-ENG-7-UNK:Execute step of Task 28000000002 : Interruption Handler: Select Resource (--Triggering Contact-- from CSQ)
%MIVR-SS_RM-7-UNK:Task: 28000000002 Interrupting the task has been successful.
Telephony events for consult transfer of call from CTI Port (2508) to agent (2302) – first CTI Port gets put on Hold
%MIVR-SS_TEL-7-UNK:CallID:0 MediaId:1007/2 Task:28000000002, transfer(2302, 12000, ACKNOWLEDGED)
%MIVR-SS_TEL-7-UNK:CallID:0 MediaId:1007/2 Task:28000000002 com.cisco.jtapi.CiscoRTPInputStoppedEvImpl received
%MIVR-SS_TEL-7-UNK:CallID:0 MediaId:1007/2 Task:28000000002 com.cisco.jtapi.CiscoRTPOutputStoppedEvImpl received
%MIVR-SS_TEL-7-UNK:CallID:0 MediaId:1007/2 Task:28000000002 Got CallCtlConnDialingEv, events on the AddressCallObserver.
%MIVR-SS_TEL-7-UNK:CallID:0 MediaId:1007/2 Task:28000000002, TerminalConnection to Terminal: UCCX_2508 is HELD
%MIVR-SS_TEL-7-UNK:CallID:0 MediaId:1007/2 Task:28000000002 Got ConnInProgressEv, CallCtlConnOfferedEv, events on the AddressCallObserver.
Telephony events for consult transfer with call alerting (ringing) at agent (2302)
%MIVR-ICD_CTI-7-UNK:EventHandler: posting {CALL_DELIVERED_EVENT: Socket:Socket: null monitoredDeviceDN:2302, connectionCallID: 33555439, ...
%MIVR-SS_TEL-7-UNK:CallID:0 MediaId:1007/2 Task:28000000002 Got ConnAlertingEv, CallCtlConnAlertingEv, events on the AddressCallObserver.
%MIVR-SS_TEL-7-UNK:Call.transferStarted(2302) JTAPICallContact[id=0,implId=1007/2,state=STATE_ANSWERED_IDX,inbound=true, ...
Log Analysis
Call Delivery to Agent—Log Analysis
Agent (2302) answers the call
%MIVR-ICD_CTI-7-UNK:ClientConnMgr: Processing msg on socket:Socket[addr=192.168.2.102,port=53461,localport=12028] Msg is {length=28
type=ANSWER_CALL_REQ,invokeId=23, peripheraid: 1, connectioncallid: 33555439, connectiondeviceidtype: 0, connectiondeviceid: 2302,
agentinstrument: 2302 }
%MIVR-SS_TEL-7-UNK:CallID:0 MediaId:1007/2 Task:28000000002 Got ConnConnectedEv, CallCtlConnEstablishedEv, events on the AddressCallObserver.
Agent (2302) answers the call so CCX instructs to complete the consult transfer
%MIVR-SS_TEL-7-UNK:CallID:0 MediaId:1007/2 Task:28000000002, transfer(2302, consultCall)
%MIVR-SS_TEL-7-UNK:CallID:0 MediaId:1007/2 Task:28000000002 Got CiscoTermConnSelectChangedEv UCCX_2508, events on the AddressCallObserver.
%MIVR-SS_TEL-7-UNK:CallID:0 MediaId:1007/2 Task:28000000002 Got CiscoTransferStartEv, events on the AddressCallObserver.
%MIVR-SS_TEL-7-UNK:CallID:0 MediaId:1007/2 Task:28000000002 Got ConnDisconnectedEv, CallCtlConnDisconnectedEv, events on the AddressCallObserver.
%MIVR-SS_TEL-7-UNK:Call.transferAnswered(2302) JTAPICallContact[id=0,implId=1007/2,state=STATE_ANSWERED_IDX,inbound=true, ...
%MIVR-SS_TEL-7-UNK:CallID:0 MediaId:1007/2 Task:28000000002 Got CiscoTransferEndEv, events on the AddressCallObserver.
%MIVR-SS_TEL-7-UNK:Call.transferring(2302) JTAPICallContact[id=0,implId=1007/2,state=STATE_ANSWERED_IDX,inbound=true, ...
%MIVR-SS_TEL-7-UNK:CallID:0 MediaId:1007/2 Task:28000000002 Got TermConnDroppedEv UCCX_2508, CallCtlTermConnDroppedEv UCCX_2508,
ConnDisconnectedEv, CallCtlConnDisconnectedEv, CallInvalidEv, events on the AddressCallObserver.
%MIVR-SS_TEL-7-UNK:CallID:0 MediaId:1007/2 Task:28000000002 Got CallObservationEndedEv, events on the AddressCallObserver.
%MIVR-SS_TEL-7-UNK:CallID:0 MediaId:1007/2 Task:28000000002, got CiscoTransferEndEv event on the AddressCallObserver, posting it to InCallObserver. ...
%MIVR-SS_TEL-7-UNK:CallID:0 MediaId:1007/2 Task:28000000002 Got CallObservationEndedEv, events on the AddressCallObserver.
%MIVR-SS_TEL-7-UNK:CallID:0 MediaId:1007/2 Task:28000000002, got TransferStart event, and final call 1007/2 is the same as the consult call, Transferred call in
event (P1-jtapi_1) GCID=(2,1008)->INVALID
%MIVR-SS_TEL-7-UNK:CallID:0 MediaId:1007/2 Task:28000000002 gets TermConnDroppedEv, meta code:134, cause code:100
%MIVR-SS_TEL-7-UNK:Call.abandoned() - transferring JTAPICallContact[id=0,implId=1007/2,state=STATE_ANSWERED_IDX,inbound=true, ...
%MIVR-SS_TEL-7-UNK:Call.transferred(2302) - transferring JTAPICallContact[id=0,implId=1007/2,state=STATE_TRANSFERRED_IDX,inbound=true, ...
%MIVR-SS_TEL-7-UNK:CallID:0 MediaId:1007/2 Task:28000000002, got TransferEnd event, finalCall: (P1-jtapi_1) GCID=(2,1007)->INVALID, consultCall: (P1-jtapi_1)
GCID=(2,1007)->INVALID, success: true, transferStart: false
Script workflow is ended and Agent (2302) is set to IN SESSION (Talking) state, relevant CTI messages are posted
%MIVR-ENG-7-UNK:Execute step of Task 28000000002 : End
%MIVR-SS_RM-7-UNK:Rsrc: uccxlondonagent New State:IN_SESSION Old State:RESERVED Reason code:0
%MIVR-ICD_CTI-7-UNK:EventHandler: posting {CALL_ESTABLISHED_EVENT: Socket:Socket: null monitoredDeviceDN:2302, connectionCallID: 33555439, ...
%MIVR-ICD_CTI-7-UNK:EventHandler: posting {AGENT_STATE_EVENT: Socket:Socket: null monitoredDeviceDN:2302, agentDN:2302, ...
%MIVR-ICD_CTI-7-UNK:EventHandler: posting {CALL_DEQUEUED_EVENT: Socket:Socket: null monitoredDeviceDN:null, connectionCallID: 33555439, ...
%MIVR-ICD_CTI-7-UNK:EventHandler: posting {CALL_CONNECTION_CLEARED_EVENT: Socket:Socket: null monitoredDeviceDN:2508, connectionCallID: 33555439, ...
Log Analysis
Call Delivery to Agent—What Can Go Wrong?
 Agent stuck in ―Reserved‖ state

 Agent is assigned and reserved for the call but never
receives the call.
 Typically call transfer failure—investigate
configuration for call to be routed from CTI Port to
Agent Extension
 Call appearance missing or stuck in CAD

 Typically problems with CTI events between CCX and
Agent (such as DELIVERED, ESTABLISHED,
END_CALL)
Log Analysis
Summary
 A Troubleshooting Tool
 Use troubleshooting methodology and rule out the
basics
 Log Analysis
 Practice, understand lifecycle, compare good/bad,
search for interesting strings
 Agent login, call arrival, IVR and queue treatment,

call delivery to agent
 Lifecycle, Log Analysis and typical problems
Nobody Likes a Camper…
Be Proactive – Don‟t Wait for Problems to Come Your Way
Source: ―Call of Duty 4‖ by Activision 2008

Contact Center Express
Serviceability
Serviceability
Overview of Unified CCX Serviceability
CCX System Performance Sys/App Alarms

Status Events Counters Logs
CCX Real-Time Syslog
SNMP
Real-Time Serviceability Performance Viewer
Admin Monitor
Reactive Alerts PerfMon Trace and

Logs Logs
Network
Management
System (NMS)
Proactive Alert
Notifications
NMS
Notifications
Serviceability
Real-Time System Status in CCX Serviceability Admin
Control Center – Network Services

Displays Status of CCX components
and subsystems:
SHUTDOWN-> INITIALIZING-> IN SERVICE
(PARTIAL SERVICE, OUT OF SERVICE,
NOT CONFIGURED)
Datastore Replication Servers

Displays State and Job Status of
database replication:
Connected / Active
Dropped / Timed Out
Serviceability
Real-Time System Status in Real-Time Monitoring Tool
Real Time Monitoring Tool (RTMT)

System Status shows real-time
CPU, memory, disk utilization and
alert history
Critical Services
Disk Usage
Process
CPU and Memory
Serviceability
Custom Real-Time Performance Monitoring in RTMT
Add Counters
Graph up to 6
per Cateogory Counters
Modify monitoring
Add Cateogories properties
Edit-> New Cateogory Start Logging
Create Alerts
Serviceability
Review PerfMon Logs in RTMT
 Open Perfmon logs and select counters to graph, export

as CSV to analyze raw data as you like
Serviceability
Manage Alerts in RTMT Alert Central
 System, Unified CCX and Custom Alerts
 View details, clear alerts, adjust thresholds and actions
Serviceability
Creating Custom Alerts in RTMT
 From Performance right-click on an added counter and

select ―Set Alert/Properties…‖
Serviceability
Configuring Notifications for Alerts RTMT
 In Alert Central right-click on the System, CCX or Custom
Alert and click ―Set Alert/Properties…‖
Serviceability
Real-Time Analysis of Errors in Logs in RTMT
 Syslog Viewer can be used to view critical errors and

warnings from System, Application and Security logs in
real-time
Serviceability
Configuring Tracing Levels Select Trace-> Configuration
from the Unified
Serviceability Admin menu to
configure tracing for System
services
Select Trace-> Configuration

from the Unified CCX
Serviceability Admin menu to
configure tracing for CCX
services
Serviceability
Configuring CCX Tracing Levels for Common Problems
Select Trace->Profile from the

Unified CCX Serviceability
Admin menu to enable Log
Profiles for common problems
Serviceability
Collecting Log Files in RTMT for Analysis
 In Trace and Log Central select ―Collect Files‖
Select System
Services and
Applications
Select CCX
Services and
Applications
Configure
Collect File
Options
Serviceability
Send Syslog Alarms to a Remote Syslog Server
 Select Alarm-> Configure from the Unified Serviceability

and/or Unified CCX Serviceability Admin menu
 For Platform Services the Alarm Configuration is specific to each
service and must be configured by service individually
 For CCX Services the Alarm Configuration applies to all services
Configure the
remote Syslog
Server along with
the minimum
event level
Serviceability
Alarm Definitions
 Select Alarms-> Definitions from the Unified Serviceability

Admin menu
Alarm details
The Routing List

details destinations
View alarms by components configured for each
individual Alarm
Serviceability
Simple Network Management Protocol (SNMP) Traps
 Send SNMP Traps to Network Management

System (NMS) such as CiscoWorks, HP OpenView
or other 3rd party NMS software
 Supports SNMP V1, V2C, V3 and TRAP messages (does
not support INFORM messages)
 From SNMP menu in Unified
Serviceability Admin configure
Community String (V1/V2C) or
User (V3) and the Notification
Destination
 Cisco provides Management
Information Base (MIB) files
which define the objects which
can be monitored by NMS
Serviceability
Summary
 Real-Time
 Unified CCX Admin, RTMT Performance, Syslog Viewer
 Reactive
 System and Performance Alerts, PerfMon Logs, Trace
and Log Collections
 Proactive
 Alert Notifications, sending Alarms to Syslog or to NMS
through SNMP Traps
… the End?
Source: ―Donkey Kong Country‖ by Rare 1994

Review of Session Objectives
Explore some of the key changes with

Contact Center Express Version 8.0 & 8.5
Demonstrate how to investigate problems

using available tools and logs
Share troubleshooting and serviceability tips

and best practices
Q&A
Connect with Your Peers and Cisco
 Discuss business, IT, architecture, Cisco Collaboration Community
adoption and product topics with peers and User Group
conferencing, customer care, enterprise social software, IP
communications, messaging, mobile applications, telepresence
 Interact with Cisco Product Managers,

Technical Marketing Engineers and
Services Consultants
 Learn about new product
announcements
 Join the Collaboration User Group
 Influence product direction
 Access to early product releases and info
 Get VIP perks and exclusive sessions at
Cisco Live www.cisco.com/go/joinconversation
Visit the Cisco Collaboration Booth (#1289) at Cisco Live:

Learn more about the community. Sign up for the user group.
Pick up your polo shirt and badge ribbon (user group members only).
Complete Your Online
Session Evaluation
 Receive 25 Cisco Preferred Access points for each session
evaluation you complete.
 Give us your feedback and you could win fabulous prizes.
Points are calculated on a daily basis. Winners will be notified
by email after July 22nd.
 Complete your session evaluation online now (open a browser
through our wireless network to access our portal) or visit one
of the Internet stations throughout the Convention Center.
 Don‘t forget to activate your Cisco Live and Networkers Virtual
account for access to all session materials, communities, and
on-demand and live activities throughout the year. Activate
your account at any internet station or visit
www.ciscolivevirtual.com.
Visit the Cisco Store for
Related Titles
http://theciscostores.com
98
Thank you.
Appendix A: Reference
Reference
Documentation Links
 CCX Documentation:
 http://www.cisco.com/en/US/products/sw/custcosw/ps1846/t
sd_products_support_series_home.html
 Solution Reference Network Design (SRND) Guide for

CCX 8.5(1):
 http://www.cisco.com/en/US/docs/voice_ip_comm/cust_cont
act/contact_center/crs/express_8_5/design/uccx851srnd.pdf
 CCX Administration Guide for 8.5(1):

 http://www.cisco.com/en/US/docs/voice_ip_comm/cust_cont
act/contact_center/crs/express_8_5/configuration/guide/uccx
851ag.pdf
Reference
Documentation Links
 CCX Serviceability Administration Guide for 8.5(1):

 http://www.cisco.com/en/US/docs/voice_ip_comm/cust_contact
/contact_center/crs/express_8_5/troubleshooting/guide/uccx85
1sg.pdf
 Unified Serviceability Administration Guide for 8.5(1):

/contact_center/crs/express_8_5/configuration/guide/uccx851u
s.pdf
 Real-Time Monitoring Tool (RTMT) Guide for CCX

8.5(1):
/contact_center/crs/express_8_5/configuration/guide/uccx85rtm
t.pdf
Reference
Documentation Links
 CAD Troubleshooting Guide 8.5:

 http://www.cisco.com/en/US/docs/voice_ip_comm/cust_contact/c
ontact_center/crs/express_8_5/troubleshooting/guide/cad85ccxt
g-cm.pdf
 DocWiki: Troubleshooting CCX:

 http://docwiki.cisco.com/wiki/Troubleshooting_Unified_Contact_
Center_Express
Reference
Community Resources Links
 Cisco Communities (including discussion forums):

 https://communities.cisco.com/groups/unified-contact-center-
express
 Cisco Support Community (previously NetPro forums):

 https://supportforums.cisco.com/community/netpro/collaboratio
n-voice-video/contact-center
 Contact Center User Group:

 http://www.ccusergroup.com
 CiscoCC on Twitter:
 http://www.twitter.com/CiscoCC/
Appendix B: CCX Components
CCX Components
Engine Subsystems
Component Description
Unified CM Telephony Subsystem Facilitates messages with JTAPI Client
CMT Subsystem Responsible for media termination on CTI Port
MRCP ASR Subsystem Maintains MRCP connection for ASR media channels
MRCP TTS Subsystem Maintains MRCP connection for TTS media channels
RmCm Subsystem Subsystem with multiple components: ICD_CTI, RM (resource
manager), CM (contact manager), HRDM (historical reporting data
manager), ICD_RTDM (real-time reporting data manager)
Core RTR Subsystem Responsible for providing real-time reporting data
eMail Subsystem Used for email steps in script workflow
Voice Browser Subsystem Managers voice browser functionality
HTTP Subsystem Responsible for HTTP Application Triggers
Outbound Subsystem Implements Campaign Manager and Outbound Dialer
VOIP Monitor Subsystem Connects to Desktop VoIP Monitor service for Start/Stop Monitor steps
Database Subsystem Used for database steps in script workflow
CCX Components
Desktop Services
Component Description
Cisco Desktop LDAP Monitor Monitors and replicates directory service used to store
Service service configuration data
Cisco Desktop License and Distributes licenses and monitors health of services
Resource Manager Service
Cisco Desktop Call/Chat Service Updates call/chat activity and facilitates chat messages
Cisco Desktop Sync Service Synchronizes agent/team configuration with CCX via ACMI
protocol
Cisco Desktop Enterprise Service Tracks data attached to calls
Cisco Desktop Recording and Maintains 1 day agent/team history and recordings
Statistics Service
Cisco Desktop VoIP Monitor Service Enabled VoIP monitoring by sniffing RTP packets
Cisco Desktop Recording and Enables monitored calls to be recorded and played back
Playback Service
Cisco Desktop Browser and IP IP Phone Agent CM Service and used by CAD-BE
Phone Agent Service
Cisco Desktop Agent E-Mail Service Handles/manages email as a type of contact
Appendix C:
Call Flow Lifecycle
Call Flow 6
CCX Cluster
CCX
PSTN Caller Script
7
9
3 4
1 5 10
8
2 CM Cluster 11 Agent
Voice Gateway
1&2. VG route request sent to CM 8. Agent goes into Ready state

PSTN Call CTI Route Point 9. CCX assigns call and puts Agent in
3. JTAPI route request sent to CCX Reserved state
IP Voice 4. CCX requests CM redirect call to CTI Port 10. CCX requests to consult transfer call
5. CM offers call to CCX CTI Port (ring-back) 11. Agent answers call goes to Talking
Call Control 6. Script directs CCX to answer call state
and CTI data 7. Typically prompts are played 12. Caller ends call and Agent is placed
and call is queued back into Ready state (not shown)
Appendix D:
Recommended Trace Settings
 Generally the default trace level settings are sufficient

for basic, initial troubleshooting
 The tables on the following slides provide some general
guidelines and recommendations on trace settings
generalized by broad issues or components
 Enabling excessive low-level tracing, or enabling lower
level tracing in certain situations (solution already
experiencing performance issues) on a production
system can cause performance impacting issues
 Engage with TAC for specific tracing recommendations
for the specific issue you may be facing
Component or Issue Trace Component Trace Setting
CCX Engine (from CCX Serviceability MISCELLANEOUS: ICD_CTI (Debugging)
Admin-> Trace) SUBSYSTEMS: SS_CM (Debugging), SS_RM (Debugging),
SS_RMCM (Debugging), SS_TEL (Debugging)
CM Telephony client (from CCX Enable all trace levels and all debugging levels (excluding
Serviceability Admin-> Trace) MISC_DEBUGGING), increase log file count and size from default (to
Call Control at least 100 and 2MB)
CM CallManager service (from CM Enable ―Detailed‖ trace setting and SDL trace
Unified Serviceability-> Trace)
CM CTIManager service (from CM Enable ―Detailed‖ trace setting and SDL trace
Unified Serviceability-> Trace)
CCX Administration (from CCX LIBRARIES: LIB_AXL (Debugging, XDebugging1), LIB_CFG
CCX Admin and Serviceability Admin-> Trace) (Debuging, XDebugging1), LIB_JASPER (Debugging, XDebugging1-
Serviceability Admin 3)
MISCELLANEOUS: ADM_CFG (Debugging, XDebugging1)
CCX Cluster View Daemon (from CCX MISCELLANEOUS: CVD (Debugging), CLUSTER_MGR (Debugging)
High Availability and Serviceability Admin-> Trace) Enabling Xdebugging on above components can cause performance
Failover issues
CCX Engine (from CCX Serviceability MISCELLANEOUS: ICD_CTI (Debugging) (enable XDebugging1 for
Admin-> Trace) issues with CAD CTI, or Xdebugging4 for 3rd party call control issues
CCX Resource/Contact like transfer/conference)
Issues SUBSYSTEMS: SS_CM (Debugging, XDebugging1), SS_RM
(Debugging, XDebugging1), SS_RMCM (Debugging), SS_TEL
(Debugging, XDebugging1)
Automatic Speech CCX Engine (from CCX Serviceability SUBSYSTEMS: SS_MRCP_ASR (Debugging, XDebugging1),
Recognition and/or Text Admin-> Trace) SS_MRCP_TTS (Debugging, XDebugging1)
To Speech
Platform DRF Logs saved to platform: /var/log/active/platform/drf/log/ in format
<datetime>_<component>_b.log for backup logs and
<datetime>_<component>_r.log for restore logs. Collect from RTMT
Disaster Recovery
by selecting ―Cisco Unified CCX DRF‖
System DRF Master and Local stored at /var/log/active/platform/drf/trace/ and
collect from RTMT by selecting ―Cisco DRF Local‖ or ―Cisco DRF
Master‖
CCX Administration (from CCX LIBRARIES: LIB_LICENSING (Debugging, XDebugging1)
Serviceability Admin-> Trace)
Licensing
CCX Cluster View Daemon (from CCX LIBRARIES: LIB_LICENSING (Debugging, XDebugging1)
CAD client Open: C:\Program Files\Cisco\Desktop\config\Agent.cfg

Cisco Agent Desktop Change ―Threshold=OFF‖ to ―Threshold=TRACE‖
Increase ―Files‖ and ―Size‖ as needed
CSD client Open: C:\Program Files\Cisco\Desktop\config\Supervisor.cfg

Cisco Supervisor Change ―Threshold=OFF‖ to ―Threshold=TRACE‖
Desktop Increase ―Files‖ and ―Size‖ as needed
CCX Engine (from CCX Serviceability MISCELLANEOUS: ICD_CTI (Debugging), XDebugging4)

Admin-> Trace) SUBSYSTEMS: SS_CM (Debugging, XDebugging3), SS_RM
(Debugging, XDebugging3), SS_RMCM (Debugging, XDebugging3)
IP Phone Agent
CCX Desktop Services (from CCX Cisco Browser and IP Phone Agent Service set to Debugging 2
CCX Engine (from CCX Serviceability MISCELLANEOUS: ICD_CTI (Debugging, XDebugging3)
Wrapup Code Admin-> Trace) SUBSYSTEMS: SS_CM (Debugging, XDebugging3), SS_RM
(Debugging, XDebugging3), SS_RMCM (Debugging, XDebugging3)
CCX Engine (from CCX Serviceability MANAGERS: ENG (Debugging)

Admin-> Trace) STEPS: STEP_<Step Type> (Debugging)
Script Workflow ENG debugging will trace workflow steps as executed, enable specific
step debugging for further detail as required
CCX Engine (from CCX Serviceability MANAGERS: ENG (Debugging)
HTTP Admin-> Trace) SUBSYSTEMS: SS_HTTP (Debugging)
CCX Engine (from CCX Serviceability MISCELLANEOUS: CRA_HRDM (Debugging, XDebugging1)

Historical Reporting Admin-> Trace) SUBSYSTEMS: SS_CM (Debugging), SS_RM (Debugging,
XDebugging1), SS_TEL (Debugging)
HRC client Open: C:\Program Files\Cisco UCCX Historical Reports\hrcConfig.ini

Historical Reporting Modify the log level to equal 5
Client
HRC client Open: C:\Program Files\Cisco UCCX Historical

Historical Reporting Reports\Scheduler\SCH.ini
Client Scheduler Modify the log level to equal 5
CCX Engine (from CCX Serviceability SUBSYSTEMS: SS_RM (Debugging), SS_TEL (Debugging), SS_OB
Outbound Dialer Admin-> Trace) (Debugging, XDebugging1-5)
Appendix E:
Acronym Reference
Acronym Reference
Acronym Definition
Administrative XML Layer
AXL
Unified Contact Center Express

CCX
Unified Communications Manager

CM
Computer Telephony Interface

CTI
Java Telephony Application Programming Interface

JTAPI
Real-Time Monitoring Tool

RTMT
Voice Gateway
VG

Troubleshooting Cisco Unified Contact Center Express PDF

Загружено:

Сведения о документе

Оригинальное название

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

Troubleshooting Cisco Unified Contact Center Express PDF

Загружено:

Авторское право:

Доступные форматы

Troubleshooting

Cisco Unified Contact

 Explore some of the key changes with

 Demonstrate how to investigate problems

 Share troubleshooting and serviceability tips

 Troubleshooting Contact Center Express Components

 Contact Center Express Serviceability

 Two-Server High Availability (HA) over WAN deployment

… You Need to Know the Basics

Source: ―Braid‖ by Jonathon Blow 2008

 Communications Manager (CM)

 Contact Center Express (CCX)

 Cisco Agent Desktop (CAD)

CCX Cluster Voice Gateway

CMT HTTP LRM VoIP Monitor

MRCP ASR/TTS VoIP Monitor Call/Chat Recording

RMCM Outbound Sync Browser and IPPA

Database RTR Enterprise Agent E-Mail

Database Agent Datastore Cluster View Daemon

Repository Datastore Serviceability Administration

Single CM Cluster Two-Server CM Cluster Single

 Unified CCX Components

… You Need a Strategy

“Oh…, you mean this morning about

 Was this working before? If yes, what changed?

 Clarify Problem Description

 Take a logical and methodical approach

… but You Need to Understand the Impact of

CTI Route Telephony Call Control

Unified CCX Server Unified CM Server

AXL AXL Web

Telephony Provider JTAPI User

RmCm Provider RmCm User

Telephony Triggers CTI Route Points

Call Control Group CTI Ports

 Improved Data Check Results show you exactly

 Enable debug in MADM: ADM_CFG, LIB_AXL

Admin intends to create a new RP and Line on CM

Admin intends to create a new RP and Line on CM

Recommended Steps to Recreate Telephony Triggers and Call Control Groups

 Can be caused by Telephony Data mismatch, but

 Determine whether configuration mismatch is likely, or

 Desktop Sync Service receives config update event

Unified CCX Server Unified CM Server

Config AXL Web

 Configuration Mapping of Call Flow

 Telephony Configuration and Synchronization

 Agent End User and Resource Synchronization

… Where Two Is Better than One

Source: ―Lego Batman‖ by TT Games 2008

 Utilizes a Unified Communications network

 Unified CM utilizes a clustered approach

 Unified CCX two-server High Availability cluster

HA over LAN HA over WAN

Primary CCX Server Secondary CCX Server

Primary CCX Secondary CCX Primary CCX Secondary CCX

 WAN: Both nodes have unique AXL, Unified CM

Primary Engine Fails and is detected by Primary CVD

Databases Databases Master Election initiated on Primary CVD

Master Election initiated on Secondary CVD

 Engine Failure detected CONVERGENCE_STARTED, name=Cisco Unified CCX Engine

 Master Election initiated name=Cisco Unified CCX Engine, node=1

Secondary CVD elects Secondary as Master