Вы находитесь на странице: 1из 45

eServices Troubleshooting

Denys Babyak / Roman Synyshyn


13-June-2012
Agenda

1. Overview / Architecture
2. Logging
3. Troubleshooting / Common Problems
4. Tools (Myzamir)
5. Questions

eServices Troubleshooting | June 2012 | 2


© 2012, Genesys Telecommunications Laboratories, Inc. All rights reserved.
Overview / Architecture
Overview / Architecture

What is eServices?
• eServices (called also MCR in 7.0-7.1 and Multimedia in 7.2-8.0) is a platform on which you can
assemble a coordinated suite of components that processes, manages, and archives
customer/agent interactions in the media of your choice, including social media.
Main components:
• Interaction Server (ofter referred to as Ixn)
• Universal Contact Server (UCS)
• Media Servers (E-mail Server, Chat Server, SMS Server, Social Messaging Server)
Universal Routing is not considered part of eServices because it deals with both traditional telephonic
interactions and the non-voice interactions that are handled in eServices
What's an interaction?
• It's a non-voice communication (for example, between the end-customer and agent) with a
unique id
• It has a media type (email, chat, workitem, twitter etc)
• It can be a parent to, or a child of another interaction (series of related interactions form a
thread)

eServices Troubleshooting | June 2012 | 4


© 2012, Genesys Telecommunications Laboratories, Inc. All rights reserved.
Overview / Architecture

Media Interface
IRD
Stat Server

Corporate
Mail E-mail Server URS
Corporate
Server
Mail Interaction
Server Server
Desktop
Application

Interaction
Universal DB Server
Classification
Contact
Main Server
UCS DB
Server

Archive
Ixn DB
UCS DB Knowledge
UCS Manager
Manager
Data Storage Workflow Control

eServices Troubleshooting | June 2012 | 5


© 2012, Genesys Telecommunications Laboratories, Inc. All rights reserved.
Overview / Architecture

Media Servers
• Source of inbound interactions
• Processing of outbound interactions (optional)
• Some are provided out of box (E-mail Server, Chat Server, SMS Server, Social Messaging Server)
• Custom servers can be created using SDKs

Universal Contact Server (UCS)


Interfaces a database storing:
• Contact information, such as names, addresses, phone numbers
• Contact history (previous interactions with this contact)
• Other data used in processing interactions, such as standard responses and
screening rules

eServices Troubleshooting | June 2012 | 6


© 2012, Genesys Telecommunications Laboratories, Inc. All rights reserved.
Overview / Architecture

Interaction Server
• Processes new interactions submitted by media servers
• Submits interactions to Universal Routing Server (URS) for routing
• Manages communications between URS and other eService servers
• Delivers interactions to agents
• Stops interactions
• Provides reporting to StatServer

Knowledge Management
• Classification Server (applies screening rules when triggered by a routing strategy)
• Knowledge Manager (GUI for managing Standard Response Library, Screening Rules and
Categories)

eServices Troubleshooting | June 2012 | 7


© 2012, Genesys Telecommunications Laboratories, Inc. All rights reserved.
Overview / Architecture

Business Process
A script defining an interaction processing scenario that covers the whole interaction life cycle and
reflects major steps of the interaction such as:
• Origination
• Segmentation for processing
• Processing by strategy
• Delivery to the agent desktop
• Creation of child interactions
• Termination

eServices Troubleshooting | June 2012 | 8


© 2012, Genesys Telecommunications Laboratories, Inc. All rights reserved.
Overview / Architecture

Main elements of a Business Process

Interaction Queue
• Logical parking place (persistent queue) for an interaction during processing

Workbin
• Object that holds interactions for later processing

Strategy
• Processing with or without a human target
• May receive interactions from multiple queues
• Can update interaction data, apply services, segment, prioritize etc

eServices Troubleshooting | June 2012 | 9


© 2012, Genesys Telecommunications Laboratories, Inc. All rights reserved.
Overview / Architecture

UCS Database
• Interaction (main storage for attributes common to all interactions, such as status, type,
thread id, miscellaneous timestamps, etc)
• IxnContent (stores original interaction content, for example raw email data)
• Contact (defines a contact and stores a few basic features of that contact)
• ContactAttributeMetaData (defines attributes a contact can have, as determined by
Contact Attributes under Business Attributes in Config Database)
• ContactAttribute (indicates that a contact has one of the attributes listed in the
ContactAttributeMetaData and stores its value)
• EmRoute (stores Screening Rules)
• StandardResponse (stores Standard Responses)
• Category* (stores categories tree)
• PhoneCall (stores voice-media specific data, such as ConnID)
• Emailin / EmailOut (stores email-specific data)
• Chat (stores chat-media specifc data)
See "Selected Conceptual Data Models for the UCS Database" for details

eServices Troubleshooting | June 2012 | 10


© 2012, Genesys Telecommunications Laboratories, Inc. All rights reserved.
Logging
Logging / Interaction Server

Headers:
• Application name, type and command line used to start it
• Host name and info (OS type and version)
• Timezone info
• Local and UTC time at the time of log creation
• Start time and uptime
Application name: is81pri
Application type: InteractionServer (111)
Command line: ./interaction_server -app is81pri -host roman-lx -port 2020
Host name: roman-lx
DST: TZ = 1, timeb = 0
Time zone: 0, GMT, IST
UTC time: 2012-05-25T10:20:25.049
Local time: 2012-05-25T11:20:25.049
Start time (UTC): 2012-05-25T10:20:24
Running time: 0:00:00:01
Host info: Linux, roman-lx, 2.6.18-274.3.1.el5, #1 SMP Fri Aug 26 18:49:02 EDT 2011, x86_64
File: (1) /var/SP/logs/is81pri/is81pri.log

eServices Troubleshooting | June 2012 | 12


© 2012, Genesys Telecommunications Laboratories, Inc. All rights reserved.
Logging / Interaction Server

Startup messages (1/2)


• version:
11:20:25.053 Std 05200 Genesys Interaction Server. Version 8.1.000.18

• start in backup mode


11:20:25.057 Std 04560 Warm Standby (backup) mode activated
11:20:25.057 Trc 30001 Starting in BACKUP mode, using dbid 127 for database operations

• license check out


11:20:25.077 Std 07101 Feature 'ics_multi_media_agent_seat': 600 licenses checked out

• options enumeration
11:20:25.090 Trc 06084 Configuration option set: 'settings':'default-max-submitted-per-router' = '1000'
11:20:25.090 Trc 06084 Configuration option set: 'settings':'default-max-submitted-per-strategy' = '1000':

• config objects from Tenant (scripts, strategies, queues, places, persons, business attributes etc)
11:20:25.145 Trc 21003 Loaded business process configuration, process name: 'email-processing', tenant id:
'101', state: enabled

11:20:25.177 Trc 22007 Loaded place information: name: 'Place1', tenant id: '101', state: enabled

11:20:25.198 Trc 22005 Loaded person information: employee id 'smith', tenant id: '101', name: 'Smith,
Agent', state: enabled

eServices Troubleshooting | June 2012 | 13


© 2012, Genesys Telecommunications Laboratories, Inc. All rights reserved.
Logging / Interaction Server

Startup messages (2/2)


• connecting to servers
11:20:25.623 Trc 04500 Connecting to Third Party Server 'classificationserver81' at host 'roman-lx', port 9050
11:20:25.626 Std 04503 Connected to Third Party Server 'classificationserver81' at host 'roman-lx', port 9050
11:20:25.656 Trc 04500 Connecting to Datastream consumer (Stat Server) 'statserver81' at host 'roman-lx', port 9030
11:20:25.657 Std 04503 Connected to Datastream consumer (Stat Server) 'statserver81' at host 'roman-lx', port 9030

• startup as primary
11:20:25.660 Std 05060 Application started
11:20:31.034 Trc 30002 Switching to [PRIMARY] mode
11:20:31.036 Std 04525 Port 5604 opened for listening
11:20:31.036 Std 04562 Warm Standby (Primary) mode activated

• accepting client connections:


11:20:39.749 Trc 04520 New client 18 connected
11:20:39.749 Trc 26015 Received message 'RequestRegisterClient' ('100') from client '[no name]' - Unknown:1:18,
message attributes:
attr_client_name [str] = "icon"
attr_client_type [int] = 4 [REPORTING ENGINE]
attr_ref_id [int] = 3

• database open
11:20:40.593 Std 25004 Database:'orcl:oracle', DBServer:'ixndbserver', DAP:'ixn_dap' is opened
11:20:40.593 Std 25009 DAP:'ixn_dap', DBServer version:'8.1.000.05' ('7.5.000.02' or greater required)

eServices Troubleshooting | June 2012 | 14


© 2012, Genesys Telecommunications Laboratories, Inc. All rights reserved.
Logging / Universal Contact Server

Startup
10:20:48.610 Std 05060 Application started
10:20:48.610 Std 20002 [Ucs-Main] Running 'ucs81' with Config Server running on host 'roman-lx', port 2020

• enumerating libraries versions


10:20:48.622 Std 05200 Universal Contact Server. Version 8.1.001.12 Path /opt/SP/genesys/apps/ucs81/lib/ucs.jar

• java and system info


10:20:48.721 Std 20040 [Ucs-Main] Java: version 1.6.0_26 (Sun Microsystems Inc.).
10:20:48.722 Trc 20007 [Ucs-Main] System information: 'Linux version 2.6.18-274.3.1.el5 (level unknown) - Platform
i386 (32 bits)'.

• initialization start
10:20:48.829 Trc 21007 [Ucs-Main] LCA Status changed to 'Initializing'.
10:20:48.832 Std 20031 [Ucs-Main] Load Balancing mode is : MASTER.

• reading database connection parameters, checking schema version and database structure
10:20:48.876 Std 29997 [Ucs-Main] 'ucs81' on 'ucsdap' properties: (dbcase=any) (dbengine=oracle) (port=1521)
(dbname=) (appname='ucs81' on 'ucsdap') (role=Main) (username=RMN) (interpret-prepared-statements=false)
(dbserver=roman=lx) (max-connections=100) (jdbc-debug=TRUE) (password=****) (query-timeout=20)
(jdbcurl=jdbc:oracle:thin:@uk-sup-vm1:1521/orcl)
10:20:52.293 Trc 21103 [Ucs-Main] Schema information : [id: 8.1.000.10, timestamp: 8 Feb 2012 00:00:00 GMT, version:
8.1.000.00 , description: null]

• actual startup:
10:20:58.613 Std 05061 Initialization completed
10:20:58.613 Trc 21007 [Ucs-Main] LCA Status changed to 'Running'.

eServices Troubleshooting | June 2012 | 15


© 2012, Genesys Telecommunications Laboratories, Inc. All rights reserved.
Logging / Universal Contact Server

Can't see SQL statements in UCS logs?


Set up on DAP object (used by UCS to access its database):
[settings]\interpret-prepared-statements=true

Example of UCS logging without and with interpret-prepared-statements


17:54:20.201 Dbg 29999 [SvcSrvW-3] <32 txn-11 txn-11 txn-11> Executing request : select
ContactAttribute.StrValue, ContactAttribute.Description, ContactAttribute.BinValue,
ContactAttribute.MimeType, ContactAttribute.AttributeName, ContactAttribute.ContactId,
ContactAttribute.Id, ContactAttribute.StrValueLowerCase, ContactAttribute.IsPrimary,
ContactAttribute.AttributeId, ContactAttribute.DateValue from ContactAttribute WITH
(NOLOCK) where (ContactAttribute.ContactId = ?) in transaction : txn-11 with
timeout=35

18:00:01.904 Dbg 29999 [SvcSrvW-5] <27 txn-108 txn-108 txn-108> Executing request :
select ContactAttribute.StrValue, ContactAttribute.Description,
ContactAttribute.BinValue, ContactAttribute.MimeType, ContactAttribute.AttributeName,
ContactAttribute.ContactId, ContactAttribute.Id, ContactAttribute.StrValueLowerCase,
ContactAttribute.IsPrimary, ContactAttribute.AttributeId, ContactAttribute.DateValue
from ContactAttribute WITH (NOLOCK) where (ContactAttribute.ContactId =
'0007Va6VJTFH00CF') in transaction : txn-108 with timeout=35

eServices Troubleshooting | June 2012 | 16


© 2012, Genesys Telecommunications Laboratories, Inc. All rights reserved.
Logging / E-mail Server

E-mail Server logs


• Application-level: [log] section
• Mailbox-level: each of [pop-client] and [smtp-client] sections
has enable-debug option (true/false)

eServices Troubleshooting | June 2012 | 17


© 2012, Genesys Telecommunications Laboratories, Inc. All rights reserved.
Logging / E-mail Server

Inbound email cycle


(interval defined by cycle-time option in each [pop-client] section)
• cycle start:
12:59:35.252 Trc 21621 [MsgIn-2] <pop-client> Running inbound messaging client.

• connection to server:
12:59:35.299 Trc 21633 [MsgIn-2] <pop-client> Connected to server 'IMAP' at host 'localhost', port 143.

• reading mailbox:
12:59:35.330 Trc 21009 [MsgIn-2] <pop-client> Mailbox account 'agent1'[localhost:143]: opened folder 'INBOX' with mode 'Read-Write' contains messages: 1 - unread, 1 -
new, 1 – total
12:59:35.330 Dbg 23036 [MsgIn-2] <pop-client> Looking up for 1 unread message(s)...
12:59:35.346 Dbg 23037 [MsgIn-2] <pop-client> Unread messages lookup successful. Starting to retrieve them.

• retrieving:
12:59:35.346 Trc 21010 [MsgIn-1] <pop-client> Retrieving message # 1 with size 1715 bytes.
12:59:35.346 Dbg 23040 [MsgIn-1] <pop-client> Filling record for message <000a01cd47c9$9f87ad00$99ba5687@emea.int.genesyslab.com>.
12:59:35.408 Trc 25018 [MsgIn-1] <pop-client 0007Wa7WBQ4D000N> Message # 1 stored in UCS with interaction id '0007Wa7WBQ4D000N'.
12:59:35.408 Dbg 23041 [MsgIn-1] <pop-client 0007Wa7WBQ4D000N> Retrieved message # 1 and created ixn, duration (ms): 62.
12:59:35.408 Dbg 23038 [MsgIn-2] <pop-client> Retrieval session ended: 1 msg read, 1 msg stored, at throughput 16.13 msg/s.

• cycle stop:
12:59:35.424 Dbg 23005 [MsgIn-2] <pop-client> Inbound messaging client finished.

eServices Troubleshooting | June 2012 | 18


© 2012, Genesys Telecommunications Laboratories, Inc. All rights reserved.
Logging / E-mail Server

Submitting emails to Ixn server


(interval defined by inbound-submitter-period option in [email-processing]
section)
• checking UCS database:
13:00:33.221 Dbg 23024 [InSbSchd] Looking up messages to submit to Interaction Server in database...

13:00:33.221 Dbg 23021 [InSbSchd] Found 1 matching messages in database in 0 ms


13:00:33.221 Dbg 23023 [InSbSchd] Queue size : 1 (1 new messages added).

• start submitting:
13:00:33.221 Trc 25019 [InSbW-0] <0007Wa7WBQ4D000N> Service 'EmailInternal' method 'submitInboundEmail' started.
13:00:33.236 Dbg 23027 [InSbW-0] <0007Wa7WBQ4D000N> Submitting interaction to queue 'Inbound_emails'...
13:00:33.236 Trc 21631 [InSbW-0] <0007Wa7WBQ4D000N 1369> Sending message 'RequestSubmit' to InteractionServer [w2kromans-mcr75:4420].

• cycle end:
13:00:33.361 Dbg 23028 [InSbW-0] <0007Wa7WBQ4D000N> Interaction successfully submitted to queue 'Inbound_emails'
13:00:33.361 Trc 25020 [InSbW-0] <0007Wa7WBQ4D000N> Service 'EmailInternal' method 'submitInboundEmail' ended, processing time (ms): 140.

eServices Troubleshooting | June 2012 | 19


© 2012, Genesys Telecommunications Laboratories, Inc. All rights reserved.
Troubleshooting / Common Problems
Troubleshooting / Common Problems

Tracing the interaction's lifecycle


• Has the interaction reached media server?
(example: has E-mail Server fetched the message from corporate mailbox)
• Has the media server submitted it to Interaction Server?
(look for RequestSubmit with some specific userdata unique to this interaction -
subject etc)
• Is the correct queue specified for submission, does it have a strategy linked and
loaded? Are there any additional view conditions specified?
• Has Interaction Server submitted it to URS for routing?
(look for EventRouteRequest; check if submission limits are not reached)
• Is correct StatServer (bi-directionally connected to Interaction Server) specified in
strategies?
• Are there available agents capable of handling this media type?

See Tech Tutorial #63 - Interaction Server Log Analysis and Event Flows

eServices Troubleshooting | June 2012 | 21


© 2012, Genesys Telecommunications Laboratories, Inc. All rights reserved.
Troubleshooting / Common Problems

Interactions not submitted to URS


(inbound emails not distributed to agents, outbound emails not sent - interactions are just
being placed to corresponding queues and Interaction Server doesn't submit them to URS)
The following options should be checked:
• default-max-submitted-per-router
• default-max-submitted-per-strategy

See "Interaction Submission Process" diagram in "eServices 8.1 Reference Manual“ for details

Set hidden option submitters-operation=true in [log-control] section of


Interaction Server to troubleshoot such cases (warning! this may create excessive log output)
17:48:12.750 Std 55001 Submitter for 'URS': currently submitted: 0, can submit 20; capacity limit 1000; rate
constraint: 20
17:48:12.750 Std 55001 Submitter for 'URS': found 2 strategies to submit to; details: '101:route_to_agents':
considered, currently submitted 0 vs. limit 1000; '101:send_emails': considered, currently submitted 0 vs. limit
1000
17:48:12.750 Std 55001 Submitter for 'URS': submitting to strategy 'route_to_agents' (tenant 101): number of
interactions to submit: 20; limit by strategy capacity: 1000, limit by router capacity: 20, limit by rate: 20

eServices Troubleshooting | June 2012 | 22


© 2012, Genesys Telecommunications Laboratories, Inc. All rights reserved.
Troubleshooting / Common Problems

Interaction Server <--> StatServer connectivity (1/2)


• Acting as a server and a client of each other
• StatServer Java Extensions must be enabled
[statserver]\enable-java=true
[java-config]\jvm-path=<path to .dll or .so file>
corresponding .jar files in <StatServer install>\java\ext

• Init messages in StatServer startup logs:


2012-06-11 12:54:38> Init: enable-java option is on.

Starting Java...
java version "1.4.2_12"
Java HotSpot(TM) Server VM (build 1.4.2_12-b03)
12:54:40.549 Std 10008 Loaded Java Virtual Machine from
'C:\JAVA\jre\bin\server\jvm.dll’

eServices Troubleshooting | June 2012 | 23


© 2012, Genesys Telecommunications Laboratories, Inc. All rights reserved.
Troubleshooting / Common Problems

Interaction Server <--> StatServer connectivity (2/2)


If Java is not enabled

• Interaction Server logs:

22:53:47.671 Std 04502 Cannot connect to Datastream consumer (Stat


Server) 'statserver' at host 'w2k8rmn-es81', port 7313, reason
'[connect failed]'

• StatServer logs:

2012-06-11 22:23:45> Client: Message 'DataStreamOpeningFailure'


'STRING_VALUE' 'Java Host Environment is not initialized'

eServices Troubleshooting | June 2012 | 24


© 2012, Genesys Telecommunications Laboratories, Inc. All rights reserved.
Troubleshooting / Common Problems

Example of capacity vector in StatServer logs

2012-06-07 17:32:19> Status: Capacity snapshot for place 'Place1'('Environment'), agent


'smith'('Environment') (CR='3email')
media_state current_number max_number routable
--------------- --------------- --------------- ---------------
[ R 0 3 3 ] <-- email
[ NR 0 1 0 ] <-- chat
[ R 0 1 1 ] <-- workitem
[ NR 0 1 0 ] <-- voice
[ R 0 1 1 ] <-- sms
[ R 0 1 1 ] <-- twitter
[ R 0 1 1 ] <-- fax

eServices Troubleshooting | June 2012 | 25


© 2012, Genesys Telecommunications Laboratories, Inc. All rights reserved.
Troubleshooting / Common Problems

Interactions revoked during delivery (1/2)


• Timeout defined by Interaction Server option delivering-timeout (30 sec by default)
• Option not-ready-on-invitation-timeout controls if the agent should be forced to
NotReady state (to prevent further delivery)

Event Flow:
• RequestRouteCall (to URS)

• EventInvite (to agent application)

• EventAgentInvited (reporting event)

• EventRouteUsed (to URS)

eServices Troubleshooting | June 2012 | 26


© 2012, Genesys Telecommunications Laboratories, Inc. All rights reserved.
Troubleshooting / Common Problems

Interactions revoked during delivery (2/2)

After timeout
16:08:34.033 Trc 26019 Sending message 'EventRevoked' ('134') to client '1' - Agent
application:5:984 through proxy 'GD761' - Proxy:4:984, message attributes:
<..skipped..>
attr_reason_system_name [str] = "Expired"
attr_reason_desc [str] = "Invitation expired“

16:08:34.033 Trc 26019 Sending message 'EventForcedAgentStateChange' ('190') to client


'1' - Agent application:5:984 through proxy 'GD761' - Proxy:4:984, message attributes:
<..skipped..>
attr_media_type_name [str] = "email"
attr_media_list [list, size (unpacked)=18] =
'email' [int] = 0
attr_reason_system_name [str] = "Expired"
attr_reason_desc [str] = "Invitation expired"

eServices Troubleshooting | June 2012 | 27


© 2012, Genesys Telecommunications Laboratories, Inc. All rights reserved.
Troubleshooting / Common Problems

Outbound email is not sent and loops


• Error in the logs:
19:23:00.432 Std 25100 [SvcSrvW-3] <128922
00003a3AEEFD2H21> Service 'Email' failed in method 'Send'
(ErrorCode=106 ErrorMsg=No 'To' address found.).

• This is common problem during testing stage when the same mailbox is used

• E-mail Server filters out addresses configured in pop-client sections to prevent


messages looping

• Set [email-processing]\enable-mail-loops=true

eServices Troubleshooting | June 2012 | 28


© 2012, Genesys Telecommunications Laboratories, Inc. All rights reserved.
Troubleshooting / Common Problems

Interaction Server Agent Login/Logout Errors (1/3)


Client is not logged in
• May rarely happen during a disconnect when Interaction Server has already logged out the
agent and the agent application after reconnect tries to send logout request)
17:35:21.630 Trc 26020 Distributing message 'EventAgentLogout' ('138'), message
attributes:
attr_reason_system_name [str] = "ConnectionLost"
attr_reason_desc [str] = "The agent connection has been lost“
17:38:15.733 Trc 26015 Received message 'RequestAgentLogout' ('136') from client 'IntW'
- Proxy:5248:2376, message attributes:
17:38:15.733 Trc 23016 Access denied (client is not logged in) while processing request
'RequestAgentLogout' from client 5249:2376, reference id: 6
17:38:15.733 Trc 26019 Sending message 'EventError' ('126') to client 'EM34' - Agent
application:5249:2376 through proxy 'IntW' - Proxy:5248:2376, message attributes:
attr_error_code [int] = 34
attr_error_desc [str] = "Client is not logged in“
• Similar error with RequestRemoveMedia and attr_error_desc [str] = "Not
logged in with specified media”
• Restart client application

eServices Troubleshooting | June 2012 | 29


© 2012, Genesys Telecommunications Laboratories, Inc. All rights reserved.
Troubleshooting / Common Problems

Interaction Server Agent Login/Logout Errors (2/3)


Unknown objects
• Typically happens with newly-created objects when Interaction Server doesn't have Config
Server in connections
13:56:51.993 Trc 26015 Received message 'RequestAgentLogin' ('135') from client 'GD' -
Proxy:120:680, message attributes:
attr_agent_id [str] = "42866“
13:56:51.993 Trc 23002 Unknown Person:'42866' specified in request 'RequestAgentLogin'
from client 121:680, reference id: 2
13:56:51.993 Trc 26019 Sending message 'EventError' ('126') to client '42866' - Agent
application:121:680 through proxy 'GD' - Proxy:120:680, message attributes:
attr_error_desc [str] = "Unknown agent specified in request”

• Similar message with attr_error_desc [str] = "Unknown place specified


in request"
• Restart Interaction Server

eServices Troubleshooting | June 2012 | 30


© 2012, Genesys Telecommunications Laboratories, Inc. All rights reserved.
Troubleshooting / Common Problems

Interaction Server Agent Login/Logout Errors (3/3)


Agent already logged in
13:25:38.343 Trc 26015 Received message 'RequestAgentLogin' ('135') from
client 'GD' - Proxy:7:1996, message attributes:
13:25:38.359 Trc 23005 Found that agent '5651' already logged in at place
'1' while processing request 'RequestAgentLogin' from client 13:1996,
reference id: 9
13:25:38.359 Trc 26019 Sending message 'EventError' ('126') to client '1' -
Agent application:13:1996 through proxy 'GD' - Proxy:7:1996, message
attributes:
attr_error_desc [str] = "Agent already logged in"

• Set [settings]\allow-multiple-agent-connections=true

eServices Troubleshooting | June 2012 | 31


© 2012, Genesys Telecommunications Laboratories, Inc. All rights reserved.
Troubleshooting / Common Problems

Determining delays in UCS


Search for “processing time” string in UCS

Example:

10:26:35.954 Trc 21001 [SvcSrvW-1] <11> Service 'OMInteractions' method 'CountInteractions' ended, processing time (ms): 2.
10:26:36.033 Trc 21001 [SvcSrvW-3] <13> Service 'OMContacts' method 'GetAttributes' ended, processing time (ms): 5.
10:26:36.042 Trc 21001 [SvcSrvW-2] <12> Service 'OMInteractions' method 'UpdateInteraction' ended, processing time (ms): 60.
10:26:36.122 Trc 21001 [SvcSrvW-4] <14> Service 'OMInteractions' method 'InteractionListGet' ended, processing time (ms): 28.
10:26:36.145 Trc 21001 [SvcSrvW-5] <15> Service 'OMInteractions' method 'InteractionListGet' ended, processing time (ms): 17.
10:26:36.449 Trc 21001 [SvcSrvW-6] <16> Service 'OMInteractions' method 'UpdateInteraction' ended, processing time (ms): 61.
10:26:36.469 Trc 21001 [SvcSrvW-7] <17> Service 'OMInteractions' method 'InteractionListGet' ended, processing time (ms): 13.
10:26:49.223 Trc 21001 [SvcSrvW-8] <4> Service 'Contact' method 'Identify' ended, processing time (ms): 10.

eServices Troubleshooting | June 2012 | 32


© 2012, Genesys Telecommunications Laboratories, Inc. All rights reserved.
Troubleshooting / Common Problems

Interaction are not archived / pruned in UCS


UCS prunes/archives interactions by threads. For a thread to be considered for archiving/pruning
all interactions in this thread must meet the following conditions:
• Must be stopped (attribute [UCS_DB].Interaction.Status=3);
• Interaction processing must be finished (attribute [UCS_DB].Interaction.EndDate
must not be NULL);
• Interaction attribute [UCS_DB].Interaction.EndDate must be earlier (older) than the
selected time limit for pruning/archiving.
If the interactions satisfy all conditions there are two more possible reasons:
• an email is a part of a broken thread (some email in this thread has not-existing parentid)
• there are some interactions in EmailIn/EmailOut, Chat, PhoneCall tables whose IDs
are not in the Interaction table

Both the above cases are the UCS DB corruption and that emails can be removed manually only.

eServices Troubleshooting | June 2012 | 33


© 2012, Genesys Telecommunications Laboratories, Inc. All rights reserved.
Troubleshooting / Common Problems

National characters representation


UCS supports Unicode, but other eServices or Genesys components do not (particularly
Interaction Server and URS). Interactions having Unicode characters may be corrupted. This may
include attributes handled by Interaction Server or URS such as Subject, FirstName, and
LastName. It does not include the body of the interaction, which is handled by UCS only.

Check "Language for non-Unicode programs” settings in "Regional and Language Options"
(Windows) and value of 'LANG' and 'LC_ALL' environment variables (Unix-like OS)

If this is a Java-based server (E-mail Server, UCS) try forcing -Dfile.encoding parameter
specifying the code page

Example (Knowledge Manager and Russian language):


start "Knowledge Manager" /b "%GES_HOME%\jre\bin\javaw" -Dfile.encoding=Cp1251
-Xmx1000m -classpath %CLASSPATH% -Djava.security.manager
-Djava.security.policy=.\java.policy Genesys.iknow.manager.TM_start %* 2>"%ERR%"

eServices Troubleshooting | June 2012 | 34


© 2012, Genesys Telecommunications Laboratories, Inc. All rights reserved.
Troubleshooting / Common Problems

OutOfMemory errors in Java-based servers


When processing a large amount of information (an inbound or outbound mail message with big
attachments, a thread to be archived etc) Java-based server like E-mail Server or UCS can throw
java.lang.OutOfMemoryError
Increase Xmx parameter defining maximum available memory for this JVM
• Windows: ini files (JavaEmailServerDriver.ini, ContactServerDriver.ini) or
setini.bat (older versions)
[JavaArgs]
-Xmx512M
• Solaris/Linux: shell scripts (ie. contactServer.sh):
$JAVACMD -Xmx512M

Both ESJ and UCS write current memory info into log file headers
UCS also writes memory info into logs:
17:44:42.216 Trc 29998 [MemoryMon] 53 active thread(s) - memory usage : using 45,957,744
out of 65,470,464 (70.20%) (max=533,528,576)

eServices Troubleshooting | June 2012 | 35


© 2012, Genesys Telecommunications Laboratories, Inc. All rights reserved.
Tools / Myzamir
Tools / Myzamir

Myzamir
Parses Interaction Server and URS logs
(multiple files can be loaded, depending on available workstation memory)

Sort by
• Interaction ID
• Place name
• Agent ID
• Connection ID
• Queue name
• Event name

Advanced filtering capabilities (userdata and interaction attributes)


Statistical graphs
• Wait in stack
• DB Stack size
• Processing rate
• App requests
• DB requests

eServices Troubleshooting | June 2012 | 37


© 2012, Genesys Telecommunications Laboratories, Inc. All rights reserved.
Tools / Myzamir

Myzamir - Interface

eServices Troubleshooting | June 2012 | 38


© 2012, Genesys Telecommunications Laboratories, Inc. All rights reserved.
Other resources
Other resources
Courses offered through Genesys University

Courses and Workshops


• ESRV8.1-FND eServices 8.1 Foundations (2 days)
• ESRV8.1-DEV eServices 8.1 Development (3 days)
• ESRV8.1-DPL eServices 8.1 Deployment (3 days)
• FRT7 Framework and Routing Troubleshooting Workshop 7 (3 days)

For more details on these and other classes offered by Genesys University, go
to:
• http://www.genesyslab.com/about/training/

eServices Troubleshooting | June 2012 | 40


Other resources
Tech Support Knowledgebase

Tech Support Knowledgebase


• Product Advisories
• Release Notes and Information
• Product Manuals
• White Papers
Tech Support Solution Search
• Over 30,000 Service Request Solutions for you to reference
Weekly Notifications about new or updated Product Advisories and Release
Notes
• Request on the subscriptions tab of your TS Website Profile

eServices Troubleshooting | June 2012 | 41


Other resources
Documentation

• eServices 8.1 Deployment Guide


• eServices 8.1 Reference Manual
• eServices 8.1 User’s Guide
• eServices 8.1 Selected Conceptual Data Models for the UCS Database
• Universal Routing 8.1 Business Process User’s Guide
• Universal Routing 8.1 Strategy Samples
• Genesys Events and Models Reference Manual
• Genesys Interoperability Guide
• Genesys Migration Guide
• Genesys Supported Operating Environment Reference Manual
• Genesys Licensing Guide
• Genesys Documentation Wiki (eServices Social Media Solution Guide)
http://developerzone.genesyslab.com/wiki/

eServices Troubleshooting | June 2012 | 42


Other resources
Future Tech Tutorial Webinars

Tech Tutorial #68: Network Date:07-Aug-2012


Troubleshooting

Register for these and other sessions on the Tech Tutorials page
of the Genesys Tech Support Website.
http://genesyslab.com/support/tutorials/

eServices Troubleshooting | June 2012 | 43


Other resources
Tech Tutorial Comments

If you would like to comment on:


• Usefulness of the Tech Tutorial content
• Presenter's knowledge of the subject
• Future Tech Tutorial topics
Please send email to support.webadmin@genesyslab.com

eServices Troubleshooting | June 2012 | 44


Thank you

Question
Question time
time