Академический Документы
Профессиональный Документы
Культура Документы
Contents
Applicability, Goals, and Requirements ....................................................................................................1
Best Practice Procedure and Verification .................................................................................................6
Preliminary Tasks ...............................................................................................................................6
Monitoring Concepts ....................................................................................................................6
Procedure .........................................................................................................................................11
R/3 OLTP System Monitoring.....................................................................................................11
CRM Middleware Component Monitoring ..................................................................................12
Communication Station / Admin Station Monitoring ...................................................................19
Mobile Client Monitoring.............................................................................................................24
Verification ........................................................................................................................................25
Further Information .................................................................................................................................41
Feedback and Questions........................................................................................................................51
2001 SAP AG
Best Practice: CRM (Mobile Sales) Monitoring 2
Normally, monitoring objects and their attributes are displayed in the alert monitors. However,
some interface points must be checked using generic tools and procedures. These are also
described in this document.
System Requirements
System landscape comparable to that described below in the case study.
Case Study
In this document, SAP System monitoring is discussed within the context of the mySAP CRM
Mobile Sales environment as implemented at a large German company.
The business solution consists of the following components:
mySAP CRM Mobile Sales business processes
Sales order processing
Availability of sales order processing solution
Solution landscape
Software components
Hardware components
User frontends
System availability
In this document, the mySAP CRM Mobile Sales solution is called CRM-MSA. The SAP R/3
application system is referred to as the R/3 OLTP System.
The business process flow in the CRM-MSA environment is shown on the next page.
2001 SAP AG
Best Practice: CRM (Mobile Sales) Monitoring 3
2001 SAP AG
Best Practice: CRM (Mobile Sales) Monitoring 4
CRM MW - MTS client-request picks-up the messages from the client-specific CRM
System outbound queue on the CRM Server and sends the data to the Comm. Station
mobile client
Comm Station - Protocol conversion (LDAP to RFC) Comm Station
- Forward client data to requesting mobile client Mobile Client
Mobile Client - Data from CRM System is put into the client-side inbound queue
- Mobile Client processes the data from its inbound queue
(database update and so on)
CDB = Consolidated Database on the CRM Middleware server, MTS = Message Transfer Service
(MTS client on the Mobile Client, MTS server on the Communication Station)
Solution Landscape
This section provides an overview of the solution landscape used during the analysis period:
SAP components and their release levels
Server hardware in use
User frontends
System availability is shown
Software Components
SAP Components on the CRM Middleware, Communication Station, and Mobile Client Servers
Components Release SAP Basis Kernel Components / Patch
Release Database
Release Levels
BBP/CRM
CRM Middleware 2.0B 4.6C 4.6D Oracle ABA
Basis
DCOM-Connector 2.0B - - -
Admin-Console 2.0B - - -
Mobile Client(s) 2.0B - - MS SQL
In the case study, 12 mobile clients were in productive use during the analysis period. Client
MC_REP1 is listed in this table as an example of what is installed on the other mobile clients.
The DCOM Connector and Admin Console components are installed on the same server.
2001 SAP AG
Best Practice: CRM (Mobile Sales) Monitoring 5
Hardware Components
Hardware components of the CRM-MSA server landscape
User Frontends
User frontends used during the analysis period
System Availability
The system availability plan for the CRM-MSA environment
System Name Availability Online Background Maintenance Window Peak time
Processing processing
Saturday: 20:00
R/3 OLTP Reboot+ 20:00-22:00
6 x 24 Day profile Night profile
System Offline backup at the (November)
same time with CRM
Saturday: 20:00
CRM Reboot+ 07:00-09:00 +
Online
Middleware 6 x 24 Offline backup 17:00-19:00
Process Only
System at the same time with (November)
OLTP
Comm 07.00-09:00 +
6 x 24 - - Daily backup
Station 17:00-19:00
07:00-09:00 +
Mobile Daily backup on
6 x 24 - - 17:00-19:00
client(s) CD ROM
(November)
2001 SAP AG
Best Practice: CRM (Mobile Sales) Monitoring 6
Preliminary Tasks
Before performing this Best Practice, ensure that you perform the following preliminary tasks or checks
in the system:
Complete all installation and post-installation actions and procedures including customizing
Ensure that the initial download has been successfully executed
Apply all SAP recommendations from SAP Service Sessions and any SAP recommendations
resulting from customer problem messages
Implement all current SAP Support Packages upon availability
Monitoring Concepts
Monitoring in the mySAP CRM-MSA environment can be divided into the following areas:
Monitoring the end-to-end message flow from the mobile clients to the R/3 OLTP System
Monitoring transactional RFC requests
Monitoring the queues located on the respective server systems
Monitor the R/3 Adapter components
Download objects, log file, queues, requests
Operating system performance monitoring
Database system performance monitoring
Error analysis
The following table briefly lists the CRM-MSA server components and the monitoring functions
associated with them.
2001 SAP AG
Best Practice: CRM (Mobile Sales) Monitoring 7
CRM Middleware 1. Monitor outbound queue, CRM Middleware System Mobile Client
Server
2. Monitor inbound queue, Mobile Client CRM Middleware System
3. Monitor outbound queue, CRM Middleware System R/3 OLTP System
4. Monitor inbound queue, R/3 OLTP System CRM Middleware System
5. Replication and realignment queue monitoring
6. Monitor Middleware message flow
7. Monitor flow control
8. Monitor the Middleware log
9. Monitor status of BDocs
10. Monitor initial download from R/3 OLTP System
11. Monitor R/3 Basis System
12. Monitor database
13. Monitor operating system
14. Monitor network throughput
The following graphic represents the CRM MSA server landscape. It details the monitoring interface
points on the servers.
2001 SAP AG
Business Process Monitoring in the CRM MSA Server Landscape
SMQ2
TransferService.log SMQ2
SMQ1
QMTCHECK
R/3 Application
Database
CDB
ST03
SMOHQUEUE
DB02
SM58
ST06
SM50 / SM66
SM58
Non CRM-specific SAP System monitoring transactions:
ST02, ST04, SM21, SMGW, SM13, STAT
8
R/3 OLTP System Monitoring
The SAP R/3 OLTP System is used for the processing of sales orders and as the central repository for
system data.
In its role as the lead system, the R/3 OLTP System can also be referred to as the SAP Backend
system or as the SAP Application System. In all 3 cases the same system is referred to.
After the CRM Middleware system has been installed and the initial download has been performed
successfully, system load on the R/3 OLTP System can be categorized as follows:
2001 SAP AG
Monitoring the R/3 OLTP Server
SMQ2
R/3 Application
Database
CDB
SMQ1
ST03
DB02
ST06
SM50 / SM66
SM58
SMGW
ST22
2001 SAP AG
Procedure
2001 SAP AG
R/3 OLTP System Transaction Monitoring Frequency
General Monitoring Activities / Functions Code Periods and Events
Local Work Process Overview SM50 Hourly
Monitor current state of individual work processes Daily
Ensure that enough work process capacity is Upon error
available at peak times
System-wide Work Process Overview SM66 Hourly
Similar to SM50 but for system-wide statistics Daily
Upon error
ABAP Dump Analysis ST22 Daily
Analyze aborted programs Upon error
System Log SM21 Daily
Check for general system errors Upon error
Update Errors SM13 Daily
Check for entries that have ERR status Upon error
Display Statistical Records STAT Daily
Determine the status of update records Upon error
APO_MATGUID_ACTIVE X
CRM_COND_DELTA_BLOCKSIZE 1000
CRM_DEFAULT_DESTINATION AB1_ABC
CRM_EVENT_ACTIVE X
CRM_SEND_MSG_IMMEDIATELY
These transactions and interface points are discussed in the following sections.
12
SMQ2
Monitoring the SMQR
CRM Middleware Server SMWMQUEUES
SMQ1
Log
file Flow / Process Control
Engine & other services
R/3 Application
Database
CDB
SMWMFLOW
SMO8FT
SMW01/SMW02
SMOHQUEUE
SM58
2001 SAP AG
CRM Middleware System Monitoring Transactions and Procedures
The table below lists transactions used in monitoring the CRM Middleware system.
2001 SAP AG
CRM Middleware System Monitoring Monitoring Frequency
Queue Monitoring Activities / Functions Type Periods and Events
Scheduler Status SMQR Hourly
This transaction runs the scheduler for checking Daily
the inbound queues on the CRM Middleware
Server Upon error
Ensure that all inbound queues are registered (type
R).
Register them, if necessary, using transaction
SMQR
If a queue is registered and has not been
processed for a long time the administrator has to
check the reason for it.
Possible middleware scheduler problems may
exist.
Check maximum processing time of the inbound
queues
Set the maximum processing time of a queue
("Max-time") to 5 seconds for all queues during
normal operations
Replication Queues Monitor SMOHQUEUE Hourly
Displays information about the status and contents Daily
of the replication and realignment queues in the
clients defined in the CRM Middleware System
The EXTRACT queue shows to which clients a
specific BDoc will be distributed
To check status of the queues
o Deselect 'Display current client only' flag
o Press 'REFRESH' button
Search for queues with
o Status = Hold
o # of entries > 0 and a
o flash icon button in the right-most column
Correct the error situation indicated for the
indicated queues
15
CRM Middleware System Monitoring Monitoring Frequency
Message Flow Monitoring Activities / Functions Type Periods and Events
Middleware Message Server flow monitoring SMWMFLOW Daily
Collects statistical data about the workload on the Upon error
middleware server caused by BDocs
Use this as a start-point for analysis of performance
problems
The message flow is a central function to monitor
BDocs in the Middleware System
Display statistics on the message flow within CRM
Middleware
Ensure that the middleware message flow statistics
are switched on
Ensure that the middleware kernel application
statistics are switched on ("MW" in column type
and "Middleware Message Hub Statistic" in column
Text, flag "act" is set)
16
CRM Middleware System Monitoring Monitoring Frequency
R/3 Adapter / Download Monitoring Activities / Type Periods and Events
Functions
Monitor Download Status R3AM1 Upon errors after
Checks whether the initial download was initial download
completed successfully. Dependent on
Object status and actions: object status
o RED: refer to SAP Note 309734 for initial
download problem analysis
o YELLOW: initial download must still be
done
o GREEN: OK
Logfile Communications OLTP/Middleware R3AM2 Daily
Display of the log files for communication between
the OLTP System and the CRM System
Recommended to use instead of transaction
SMO8FT
Monitor Queue Status R3AM3 Monitor the
Monitors the Delta Download queues on OLTP outbound queue on
system the R/3 OLTP
System directly on
the OLTP via smq1
Monitor Request R3AM5 Upon error, if
Used in exceptional cases to bring the database databases are not
into a consistent state after a data lost in the CDB consistent and
request from OLTP
is necessary
17
CRM Middleware System Monitoring Monitoring Frequency
General Monitoring Activities / Functions Type Periods and Events
Database Performance Analysis ST04 Hourly
Monitor database statistics Daily
ABAP Dump Analysis ST22 Daily
Analyze aborted programs Upon error
System Log SM21 Hourly
Check for general system errors Upon error
Gateway Monitor / Connection List SMGW Upon error
Monitor the active connections to other servers
Display gateway trace
Check the SAP return code and CPI-C return
code values for errors
Update Errors SM13 Hourly
Check for entries that have ERR status Upon error
Display Statistical Records STAT Upon error
Determine the status of update records
18
Communication Station / Admin Station Monitoring
The Communication Station (Comm Station) is the link between the CRM Middleware server and the
Mobile Clients. The Comm Station is the server where the DCOM Connector component is installed.
The SAP system statistic collector daemons, SAPOSCOL and RFCOSCOL, run on the Comm Station
and gather hardware resource consumption data. Complementary programs run on the CRM
Middleware server and collect and display statistical data.
The Administration Station (Admin Station) is used for administering the Mobile Clients. Tasks such as
registering each mobile client and assigning site IDs and queue names are done on the Admin
Station. The Admin Station component is not processing-intensive and monitoring is not a primary
concern with it.
The following diagram shows the areas of monitoring in respect to the Communication Station.
19
Monitoring the
Communication Station
DCOM Connector Monitor
TransferService.log
R/3 Application
Database
MS Windows NT
Performance Monitor
CDB
2001 SAP AG
DCOM Connector Monitor
Use the following menu path to access the DCOM Connector component monitor: Start > Programs >
Middleware > DCOM Connector. Start the application and choose Monitor. The following tables
explain the different areas of the monitor.
2001 SAP AG
The default file path is as follows:
C:\WINNT\TransferService.log
The TransferService.log file supplies the following data for each session. This data is stored and can
be displayed in text format on the local Comm Station server.
22
Windows NT Performance Monitor
The Microsoft Windows Performance Monitor (PERFMON) is a performance-monitoring tool on
Windows NT and Windows 2000 systems.
PERFMON can be used on the Communication Station to analyze performance problems due to:
Total response time
Network time
Browser generation time
When analyzing network conditions, PERFMON should be run locally on the Comm. Station rather
than from a remote server. When running PERFMON from a remote server, network statistics can be
affected due to the network load caused by PERFMON itself. Other network-based applications, such
as PCAnywhere, also create a significant network load that may invalidate any network statistics
measured while connected.
Handling procedure for Performance Monitor:
Verify that the Performance Monitor is installed
Set up the counters and the log file (modify/set the log file and chart settings)
Ensure that no other services or programs are running that may impact the measurement (such as
programs causing network or CPU load).
Perform the measurement
Extract the relevant counters (export them to a file)
Calculate the relevant quantities
Interpret the results
For further details, go to www.microsoft.com > Support Knowledgebase and see the White Paper
Measuring performance-relevant data using PERFMON on Windows NT.
For a detailed description of the use of the MS Performance Monitor, see SAP Note 203845.
In Windows NT, to start PERFMON, choose Start > Run and enter PERFMON.
Set up PERFMON as follows.
1. Measure the CPU utilization on the server or PC.
From the PERFMON menu, choose Edit > Add to Chart / Add Item
Select in PERFMON: Object, Processor, Counter, % Processor Time
2. Determine the size of dataset transferred for analysis.
Select in PERFMON: Object, Network Interface, Counter, Bytes Received/sec
Note: Depending on the network connection, the name of the category and item of the above-
mentioned examples may differ. When selecting the network interface counter it is possible that
various instances (= lines) are offered. If you are not sure on which network line the traffic takes
place, choose all instances. You may have to start processes at the NT level monitoring network
traffic.
3. To start your analysis, monitor during the peak time when Mobile Clients are uploading to the CRM
Middleware System. To save the performance data and transfer it into a spreadsheet program,
choose File > Export Chart.
4. Evaluate your analysis.
23
Mobile Client Monitoring
QMTCHECK.EXE
The QMTCheck program monitors the connection between the Mobile Client and the Communication
Station.
Data bound from the CRM outbound queue to a specific mobile client is copied to the inbound queue
of that mobile client.
Data from the Mobile Client outbound queue is copied to the inbound queue on the CRM system.
The Inbound and Outbound Queues of the Mobile Client can only be monitored on the mobile client
using the program QMTCHECK.EXE
Default file path: C:\Program Files\sap\mobile\bin\qmtcheck.exe
Usage
To connect to the client's database, click Open DB Connection.
To update the Mobile Client Queue status, click Update View.
Functions
Send and receive messages to the CRM Middleware system
Shows status of data transmissions
Restart inbound and outbound queues processing
When the ConnTrans program is started, or restarted, previously unprocessed inbound and outbound
queue entries will be processed.
After connection to the CRM system the outbound queue empties. Entries remain in the outbound
queue until they are successfully copied to the inbound queue of the CRM system. Entries that remain
in the client outbound queue after ConnTrans was started have not been processed.
The inbound queue of the mobile client should also be empty after running ConnTrans. Entries that
remain in the client inbound queue after ConnTrans was started have not been processed.
Both the Mobile Client outbound and inbound queues should be empty within a relatively short time
after ConnTrans has been started.
Usage
On the Mobile Client, choose Start > Programs > SAP CRM Mobile Client > ConnTrans.
24
TransferService.log file
File logs the communications between the Mobile Client and the Comm Station.
The format of the TransferService.log file on the Mobile Client is not so clearly self-documenting as the
log file on the Comm Station.
In most cases the client log file should not be analyzed until after analyzing the Comm. Station
TransferService.log file. Although both files have the same name, their formats are very different. The
log file on the Comm Station is documented in the section of this document about the Comm Station.
When diagnosing SAP-related communications problems on the Mobile Client, first look at the log file
from the Comm Station.
Verification
After you execute this Best Practice, you can perform monitoring in your mySAP CRM Mobile Sales
solution using the transactions described in this section.
Many CRM-specific transactions are not available in the R/3 OLTP System. Those common to both the
R/3 OLTP System and to the CRM Middleware System are described together. Those that are handled
or interpreted differently are explained separately.
25
Queues to mobile clients start with CRM_SITE_
Queues for the OLTP system start with R3A..., number of these queues depend on the
business logic and Customizing settings
The OLTP system should always be connected (except during the downtime for backup).
The serialized R/3 Adapter relevant queues all start with
R3AI<object name>
R3AD<object name><key>
R3AR<object name>
When, in the OLTP system, mass updates are done, e.g. application data updates, then the data gets
shipped via the outbound queue to the CRM Middleware system.
This data is controlled via the qRFC settings.
To see the current qRFC version in SMQ1, choose Information > Version.
For a detailed description of qRFC, see SAP Notes 166096 and 193515.
Usage
All queues named CRM_SITE_* are BDocs data from Middleware going to the client
All queues named R3A* are BAPIs destined for the OLTP system
CRM SMQ1: R3D* : BAPI data going from the Middleware system to the OLTP system
OLTP SMQ1: R3D* : BAPI data going from the OLTP system to the Middleware system
Handling Procedure
If you delete a queue that is in use between a mobile client and CRM, this will lead to an
inconsistency between CRM and the mobile client.
In this case, you must bring the client back into a consistent state. In some cases, you can do this by
extracting the data from the Middleware system and sending it to the client. This restores the client
state as it is known on the Middleware system. Data that was not yet successfully transferred from the
client to the Middleware must be manually reconstructed. Local data that is not normally sent to the
Middleware system may be also lost and you may need to reconstruct it manually.
If you accidentally delete data from a queue that is not assigned to a mobile client, data may be lost
and cannot be recovered (for example, in APO, BW, OLTP).
In the case of an error, first try to solve the problem. If all other attempts to fix the problem fail, you
may need to delete an entry in a queue.
In these cases, you must bring the different databases (OLTP, CRM Middleware Server, CRM Client)
manually into a consistent state. To do so, you must be familiar with the technical and business logic of
the process involved (if you are not sure what to do, contact SAP Support).
Problems and errors can occur during data exchange (such as program crash, lost connection
between OLTP System and CRM Middleware Server, and so on).
Queues for OLTP should not wait too long.
Possible causes for waiting: error of first BAPI in queue, resource bottlenecks in OLTP, logical
sequence.
Check the status. If it is SYSFAIL or CPICERR, an error has occurred for the corresponding queue
(most likely on the CRM Middleware Server). Double-click on the queue name and look at the status
text of the first entry in the queue: you will find a short error message describing the error. After
correcting the error, you can restart the queue.
26
Outbound Queue Tables
Queue: TRFCQOUT
Data: ARFCSDATA
Status: ARFCSSTATE
Handling Procedure
If you recognize many entries in one or more queues, take care of the following points.
The corresponding client of the queue has not been connected to the CRM Middleware Server for
a long time. You can check this in the column 1. Date. In this example the client was not connected
to the system since 19. October. You should avoid this situation. If there are many queues with a
large amount of data inside it, this can lead to a rapid grow of communication tables (e.g.
ARFCSDATA, ) and have an impact on system performance. Normally the client should be
connected to the CRM Server regularly. In exceptional cases at least every two weeks.
To reduce the amount of data the subscriptions and publications for a client should as small as
possible necessary from the business point of view.
For all other queues not assigned to a client errors should be monitored hourly at least several
times per day. Errors should be resolved and the queue restarted afterwards.
Handling Procedure
An error has occurred, if during an object download with status Running (light yellow) the date, time
and the number of blocks remain constant over the time (about five minutes). Normally no queues
should be build up here.
General procedure: In the case of stopped queues first search for the error causes and then fix them.
Following are listed some possible error situations.
If the status of the queue is SYSFAIL or CPICERR, an error has occurred.
- By double clicking on the queue name and looking at the status text of the first entry in the
queue, you will find a short error message describing the error reason. After the correction of
the error, you can restart the queue.
28
- Check the short dumps (transaction ST22) on the CRM Middleware Server and the OLTP
system. An error in the queue data processing in generally generates a short dump. Here you
can find information about the cause and location of the error.
- Check further information about errors in the system log (transaction SM21) on the CRM
Middleware Server and on the R/3 OLTP side. Also in the CRM Middleware Server Log.
The input queue on the CRM Middleware Server has status Ready, but the number of entries do
not decrease:
Transactions cannot be performed because the system is overloaded. If enough systems
resources are available, start the SMQ1 transaction menu Execute LUWs and check queue
registration in SMQR.
If the queue cannot be activated again, check the application log for possible application error
messages.
Data arrives from the OLTP system but is still hanging in the inbound queue of the CRM
Middleware Server:
- Check that the scheduler is set to active on the Middleware server (transaction SMQR)
- Follow-up problems can be found in the tRFC queue (transaction SMQ1) or in case of short
dumps in transaction ST22.
The inbound queue stops after each object:
Check whether a breakpoint exist in the SMOFPARSFA parameter file. If so (parameter name
BREAK-POINT), please delete them.
If no delta data arrived from the R/3 OLTP (no data in the inbound queue of the CRM Middleware
Server), or the Delta queue has the status STOP you can proceed as follows:
At least one object has download status Running or Abort and has not finished. Wait until the
object is finished or you have to force termination (development support needed).
See the subsection about SMQ1 in the section about OLTP.
Usage
Menu path: Middleware > Monitoring > Queues > Replication Queues
Transaction SMOHQUEUE
29
Field Names Descriptions
Name Name of queue:
AC_EXTRACT Represents those extract jobs triggered directly from the Admin
Console, for example for a data reload to a mobile client to
recover the local SQL database
EXTRACT Contains extract jobs broken down by BDocs (the jobs are
created by realignment)
REALIGN Contains realignment jobs for individual BDocs whose
responsibilities must be checked
SUBCHECK Contains subscription checker jobs (these are generated in the
administration console, if a subscription is changed)
No. entries Current number of entries (messages) in the queue
Number of queue
entries
AC_Extract Released 0
Handling procedure
Normal condition: All lights green or yellow, number of queue entries is decreasing.
If an error occurs during processing the Queue demon automatically sets the queue to hold. An
error symbol appears in the last column. You can select this to display the error message. When
you release the queue again, the error status is reset.
If you release a queue after the Queue demon has started, the status of the queue changes to
Running, provided it contains entries for processing. If all entries have been processed (number of
entries is zero), the status changes to Released again.
If the number of entries in a queue remains constant for a period of time, it may be necessary to trigger
the processing jobs manually by double-clicking the green flag corresponding to the queue.
If a queue contains entries, you can display details of the individual entries (such as the number of
sites affected) by clicking on the number of entries. You can delete individual entries in this display, by
selecting the checkbox in the first column of the record and choosing Delete. To update the information
choose Refresh.
30
Try to avoid deleting an entry, because this leads to inconsistency of the local database of the mobile
client.
31
Flow itself
Therefore the sum of the total response times of all services can be significant smaller than the
total response time of the BDoc type you have chosen before.
2. In SMWMFLOW, choose Last minutes workload
Here you can get statistics about the current workload of the message flow in your system (for
BDoc types and for CRM Middleware services).
To display statistics for the Middleware services, you can also choose Goto > Service statistics. In
the service statistics you can see Detail statistics for individual services as well as the workload for
individual services per BDoc type.
In all detail statistics, to display individual statistics for the selected BDoc type or service, choose
Single records.
Alternatively, in SMWMFLOW, choose Last minutes workload > Per service
Here you can display the workload for individual BDoc types per service.
This overview does not include the processing time for the following:
Inbound adapter
OLTP inbound adapter (mapping and keygen)
Flow itself
Therefore the sum of the total response times of all services can be significant smaller than the
total response time of the BDoc type you have chosen before.
32
Waiting Times
Number of BDocs in the queue as well as the total, average and maximum queue time of the
BDocs in these queues.
You can branch from each profile into other profiles and you can always change the time period
you selected at the beginning by choosing Change period. The following screens display the
workload data compressed per selected profile, that is by BDoc type, site, queue, or time.
In SMWMFLOW, choose Message Flow Statistics > Workload from Database > BDoc type profile
The table shows you workload data compressed per selected profile, that is by BDoc type.
In SMWMFLOW, choose Message Flow Statistics > Workload from Database > Site Profile
The table shows you workload data compressed per selected profile, that is by Site.
In SMWMFLOW, choose Message Flow Statistics > Workload from Database > Queue Profile
The table shows you workload data compressed per selected profile, that is by Queue.
In SMWMFLOW, choose Message Flow Statistics > Workload from Database > Time Profile
In this screen you can display the processed BDocs within different time periods (per hour, day, or
week).
If you selected one day as the time period, By time refers to hours.
If you selected a week as the time period, By time refers to days.
2. In SMWMFLOW, choose Message Flow Statistics > Single statistics records
The single statistics always relate to one day and always to the current R/3 instance (setting in
field Instance).
The header data shows you which display you are currently in and what your selection criteria
were.
The table shows the single records that the statistics refer to. For each single record the BDoc
type, client, site, queue and name of the Middleware Server are displayed, as well as the retention
time of the corresponding BDoc on the Middleware Server.
From this display you can branch into a profile and you can always change the time period you
selected at the beginning by choosing Change period.
To display the Middleware log for the services that have processed the message, choose Edit >
Message > Log.
To display the BDoc body segments using the structure editor, choose Edit > Message > Display
body.
To display errors for a BDoc, choose Edit > Message > Display errors.
To delete messages, resume processing the message with the last service, or retry processing,
choose Edit > Message > Error Handler.
To release messages locked in a qRFC queue, choose Edit > Message > Release queue.
This may be necessary in exceptional cases when the R/3 OLTP Adapter cannot be called so the
queue is locked.
Usage
Menu path: Middleware > Monitoring > Queues > Display Queue Information
Transaction SMWMQUEUES
Usage
Menu path: Middleware > Monitoring > R/3 Adapter > Display Download Queues
34
Transaction R3AM3
Usage
Menu path: Middleware > R/3 Adapter > Synchronization Load > Monitor Request
Transaction R3AM5
Usage
Menu path: Middleware >
Transaction SMO8FDC
Usage
Menu: Middleware > Monitoring > Flow Control > Middleware log.
Transaction SMO8FA1
Alternatively, you can view the log from the Flow Control Trace list screen.
Recommendation: Use R3AM2 instead of SMO8FA1 because the selectivity level provides better
top-down monitoring support.
Information Level:
A (abort)
E (errors)
W (warning)
Different info levels
Different trace level
Message category (log of R/3 Adapter, flow control MW server, CDB service MW server, replication
MW server)
Note: R3AM2 reads the SMO8_LOG table. Usage of R3AM2 is recommended rather than SMO8FA1.
Usage
Menu path: Middleware > R/3 Adapter > Initial Download > Monitoring > Display Log File
Transaction R3AM2
Handling Procedure
If background processing scheduler status is ACTIVE or WAITING, select Refresh (F9) at short
intervals (approx. 1 second) and monitor the background processing scheduler status as well as date
and time of the field Last update(every 2 minutes).
36
If the status of the scheduler is occasionally WAITING, and the date, time in field Last update (every 2
minutes) changes every few seconds without a decreasing number of queue entries in SMQ2 this
indicates that the qRFC scheduler gets not enough resources. The qRFC scheduler may then not start
any asynchronous RFC calls (all dialog work processes are blocked) with the result that the qRFC
inbound queues are not processed.
You can check whether the system can provide enough resources by using transaction SM50. A few
dialog work processes should remain in status waiting. Otherwise the system is overloaded.
Usage
Menu path: Middleware > Monitoring > Transactional RFC
Transaction SM58
Handling Procedure
When error messages appear in the SM58 display, the error condition(s) must be resolved before re-
execution of the tRFC procedure.
When the problem is corrected, the tRFC should be manually re-executed / re-scheduled.
If the error is fixed, the entry does not appear in the SM58 display.
Do not delete an entry before resolving the problem because this will lead to a data inconsistency.
37
The Workload Monitor is used to analyze statistical data from the R/3 kernel. When analyzing the
performance of a system, you should normally start by analyzing the workload statistics. For example,
you can display the totals for all instances and compare the performance of individual instances over
specific periods of time.
You can use the workload monitor to display the:
Number of configured instances for each SAP System
Data for all application instances, not just the one you have logged on to
Transactions used and the users that call them
Number of users working on the different instances
Performance history for recent periods for all instances
Response time distribution and resource consumption for any application server
Application server workload for today or for a recent period.
Functional description
As part of the SAP Basis, the Workload Monitor does not display any CRM specific monitoring data.
However the RFC response time statistics should be monitored relative to the RFC communications
between the R/3 OLTP System and the CRM Middleware System.
In general, it should be analyzed using the same procedures as those used in a non-CRM system
environment.
Additionally the RFC parameters and statistics should be analyzed in detail.
See also the description listed in the section about CRM Middleware monitoring.
Monitor RFCs:
Total today, last minute, last week
Goto prof RFC prof.
client (sender is caller, calls)
server (receiver is doer, answers, processes requests from other systems)
client targets
trans codes func mod.
Call time +/- = (Network time + exec time)
(Exec time + network time) = call time
Times are longer for higher data volumes
Avg. time +/- = Exec. Time / quantity
Quantity: # of times it was executed
Detail list: double-click on entry
Targets = +/- RFC destinations
ST03 click RFC Time profile
o Check to see when the heavy times are.
38
o Check wait times.
o Compare RFC response times to Dialog response times.
Example: ALE can also be included in the stats above so these have to also be considered (not only
ALE, but also all other communication applications).
Functional Description
By monitoring work processes in the R/3 System (transaction SM50) or from outside the system
(program DPMON at operating system level), you can determine the status of the work processes in
relation to the PRIV mode. If work processes are often switched to the PRIV mode, you must increase
the extended memory and/or adjust the limit for the extended memory). In this case, you must consult
with SAP.
39
You can use transaction SM66 to evaluate this information for all the work processes.
You can determine the swap space requirements at R/3 and operating system level.
Many monitoring tools are operating system dependent.
Usage
With help of this transaction you can detect long running transactions.
Monitor periodically, and also check if there are sudden performance problems.
Handling Procedures
In the case study, these transactions are evaluated during the periodic SAP EarlyWatch sessions that
are held on the system.
Expansion
RFC statistic
Time analysis
Check to see if it says no subrecords
Check RFC subrecords to see who has done what.
40
System Log SM21
Description
The System log does not display any CRM specific monitoring data.
It should be analyzed using the same procedures as those used in a non-CRM system environment.
Table name
Check these tables:
ARFCSDATA
ARFCSSTATE
TRFCQOUT
TRFCQIN (only in the CRM system)
Double-click the table entry <Tables Indexes>
Displays the table and all its indices
Select Detailed analysis
Check storage quality
If <40 (for storage quality), then the indices must be REBUILT
An index should be rebuilt at the DB level by the DBA.
Further Information
Troubleshooting
This chapter describes problems that can arise in the CRM-MSA environment and standard
procedures in handling and solving the problems.
41
Flow Diagrams
This diagram describes a way to analyze a problem encountered during the download from the R/3
OLTP System to downstream systems.
42
OLTP Download Problem:
- SMQ1 Status = RUNNING
- TIME and BLOCK NUMBER remain constant too long (>5 min.)
OLTP SM21
CRM SM21 YES?
CRM R3AM2
Errors?
Analyze error messages Fix problems
NO?
DONE
43
This diagram shows a method for analyzing a problem with the Delta Download. The chart is
continued on the next page.
OLTP System
Setup OK? NO?
CRM Inbound
Queue
Processed OK? NO?
Check queue for errors
Initial Download
finished
successfully? NO?
R3AM1
YES? Check log file
CRM
Middleware Events
activated? NO?
CRM Server
Scheduler registered
for OLTP queues? NO?
44
< Continued from the previous page >
SMOFPARSFA
table entry for
download OK? NO?
Do Filter
entries exist for
objects? YES?
Does the
Inbound Queue
stop after
YES?
each object?
Deactivate the BREAK-POINT entry in the SMOFPARSFA table
Does a
Message Flow get
generated for
objects? NO?
SMO8FD: Check
SMO8FDG: Regenerate for all objects if necessary
YES?
Problems
Solved? NO?
DONE
45
Problem Descriptions
Problem descriptions are described as follows:
Condition problem description, symptoms, etc.
Analysis possible causes of the problem, background information, etc.
Handling problem resolution procedure(s)
1) Condition
Table SMOG_REGEN is not empty. This indicates that some programming structures have not yet
been generated or that generation was not completed successfully.
Note: entries BDoc 26E6D6865C0CD411A6E6080006277FCD and BDoc BWAMESS can be ignored
if they appear in the table.
Analysis
The CRM middleware functionality is dependant on BDoc and flow programming structures. These
structures must be generated as part of the installation otherwise the server cannot perform critical
system functions.
Handling
To verify that the SMOG_REGEN table is empty proceed as follows
a. Call transaction SE16
b. Enter SMOG_REGEN as table name
c. Choose Table Contents -> Execute
d. The table should be empty
If the table SMOG_REGEN is not empty (except for the entries mentioned in the Condition section
above) proceed as follows
A. Regenerate the programming structures using the procedures listed in the post-installation
section of the CRM installation documentation
B. After regeneration, check table SMOG_REGEN for entries
C. The table should be empty
Note: If you cannot regenerate some of the structures
A. Call transaction SMOGLASTLOG
B. Enter message type E
C. Select SHOW LOG
D. Copy the information displayed into a message in SAPNet R/3 Frontend using application
component CRM-MW
2) Condition
Transaction SMQ1 shows that there are many entries in several mobile client queues. A result of this
could be that the outbound queue tables of the Middleware System become very full. This can lead to
a database standstill when the database table spaces become too full.
Analysis 1
1 or more Mobile Clients have not connected to pick-up their messages.
Handling 1
Determine why the mobile client(s) are not connecting
If the clients no longer exist, delete the references to it using the Admin Console/Station
Delete the data from the outbound queues for the non existent mobile clients
Analysis 2
If there are more subscription sites than mobile clients (e.g. for testing purposes) it is possible that the
qRFC queue has entries for nonexistent mobile clients. The publications for the nonexistent clients will
stay in the RFC queue until they are manually deleted and can impact data exchange performance
dramatically.
46
HANDLING 2
Use transaction SE37 to determine the number of subscriptions defined for the mobile clients
defined for your system (function module SMWM_PREPARE_SMOHSITEID (Single test, Execute,
click on result, SITESALL shows the number of subscriptions that are defined in the middleware)
Use the Admin Station to delete the subscriptions that are assigned to unnecessary clients
Use the Admin Station to delete the client IDs which are not needed
3) Condition
The R3AM1 transaction shows that an object from the initial download has a status of RED.
Analysis: The R3AM1 transaction shows that an object from the initial download has a status of RED.
Handling:
queue has stopped processing. The reason why the queue has stopped must be examined and
corrected.
Use SM08FT (Flow Trace) or SMW01 (IDOC Viewer) to view the error messages
Correct the error based
Using SMQ2 delete the STOP entry for the queue
Restart the queue
Use a similar procedure for subsequent errors
Do not restart the initial download from the R/3 OLTP system. The download resumes, by itself; after
the error has been corrected. See the above SAP note for a detailed handling description.
4) Condition
Lots of ARFC* jobs are listed as scheduled in the Background Job overview (SM37) on the OLTP
system. This can lead to slow system performance in the sending system, because most of the batch
processes are used in parallel.
When many of these jobs have the same remote destination this can lead to performance problems on
the remote system, if there are not enough available resources there. The currently executed LUW in
the receiving system may be terminated when it exceeds the max. allowed runtime
(rdisp/max_wprun_time). During this error the system triggers a short dump with error "TIME_OUT".
Analysis
Jobs that cannot be processed immediately by the CRM system get scheduled as ARFC jobs and are
processed asynchronously depending on system resource availability of the downstream
components (CRM, Comm. Station, mobile clients).
Batch job ARFC: SYSLOAD1<client><User>:
Is scheduled if no free dialog work processes are available on the sending system to process the
LUWs asynchronously. The affected LUW is assigned a status of SYSLOAD in the tRFC/qRFC tables.
These background jobs will then be distributed to any application server having available background
work processes. When the background work processes on the application servers become busy due
to processing these additional jobs they may not be available for the normal business processing
which they are intended for.
Use transaction SM58 regularly to find stopped LUWs and to delete them if necessary. But be careful
when deleting them in a production system to avoid inconsistencies. They can be deleted using SM58
menu: Edit Delete LUWs or Log file Reorganize.
5) Condition
Mass changes of data (creating, changing) on the OLTP system leads to reduced system performance
due to all of the dialog and batch work processes becoming occupied.
Analysis:
The outbound queue (SMQ1) of the OLTP system shows that there are between several hundred and
several thousand different queues waiting to be processed. A large amount of the changed data from
the OLTP system is intended for the delta download to the CRM Middleware Server and is therefore
stored in the outbound queue of the OLTP before being transferred to the middleware server. For
performance reasons qRFC tries to transfer as many queues as possible, as fast as possible. The
result is that many parallel queues (with different queue names) are generated.
The total number of queues that can be generated for parallel processing is limited by the possible
names that can be assigned to the queues.
Handling:
I. To reduce the possible number of parallel queues/queue names that can be created, edit the OLTP
table CRMQNAMES (transaction SE16). Search for the object name (e.g. CUSTOMER) for which you
want to reduce the number of parallel queues. For the object CUSTOMER the field CUSTOMER
(corresponds to field KUNNR of table KNA1) is used for queue name generation. This results in a
queue named e.g. R3AD_CUSTOME_<KUNNR>.
You now can truncate this field so that only this reduced part is used for queue name generation.
If you for example, change the values of the fields FLDOFFSET and LENGTH. to 7 and 3 you define
that only the last 3 characters are used in the generating of queue names. The field BAPIFLDLEN
determines the length of the field used for queue name generation.
The values for FLDOFFSET and LENGTH together may not be greater than the value used for the
BAPIFLDLEN field.
If you set the `inactive flag in the table CRMQNAMES, all data is put into one queue.
II. Beginning with PI2000.2 patch 01 you can use the parameter
CRM_MAX_QUEUE_NUMBER_DELTA in table CRMPAROLTP to reduce the number of queues. If
you set the parameter value to 5 a maximum of 5 queues are generated despite the settings in
CRMQNAMES.
III. If you are currently experiencing this problem you have the following options:
You can stop the download and activate it later (e.g. at night).
In SMQ1
- select a queue
- press F7 (glasses icon)
- menu Edit Generic queue Lock immediately. For locking the queues you can use wildcards
(e.g.R3AD*)
48
- to unlock use menu Edit Generic queue Unlock.
If you do not need the update data in the CRM Middleware Server delete the queues.
6) Condition
When large datasets are involved, the initial extract of BDocs with intelligent and dependent
distribution and the transport to the mobile clients can take a long time. These long run-times can also
lead to time-out situations.
Analysis
By default every BDoc is extracted separately, placed in the flow, sent to the client, and then read
there. This process can be optimised in most cases by applying the following handling procedures.
Handling
If you have not already done the initial load of your mobile clients, use the following recommendation
to increase the performance of the extract of the data.
Implement the bulk extract for intelligently and dependently distributed objects
Implement the parallel processing of the AC_EXTRACTS (per SAP Note 351086). Both of these
topics are described below.
After changing the parameter settings in table SMOFPARSFA, the profile RRLEX in transaction
SMOGWB must be regenerated.
Prerequisites
CRM 2.0B, SP08 and the following SAP Notes 366936, 370779, 379460, 374270, and 371234.
Additional information is in SAP Notes 361872 and 378252.
This functionality is part of CRM 2.0B SP8 or CRM 2.0C SP2. The SAP Notes are additionally
necessary.
1. Bulk Extract:
Create bulk messages to reduce the extract and transport overhead during AC_EXTRACT. This allows
up to a specified (configurable) number of instances of a Bdoc to be included in a message. Reading
these bulk messages on the clients is more efficient
Configuration: The maximum package size (number of instances of a BDoc in a message) is
determined at the time of generation using the parameter entries in table SMOFPARSFA. If the
settings are changed, the affected extract modules must be generated again (only in 2.0B, in 2.0C the
settings immediately become active).
The SMOFPARSFA table entries are as follows:
PARSFAKEY PARNAME PARNAME2 PARVAL1
RRS_COMMON MAX_PACKAGE_SIZE <empty> <Numeric value>
BULK
SUBSET
<BDoc-Name>
For PARNAME2
- Parameter BULK is used for bulk distributed messages with the specified number of instances of
Bdocs
- Parameter SUBSET is used to specify dependent and intelligent distributed BDocs
- Specify individual Bdoc-types to optimise the settings for individual BDoc types
- This field should not be left empty. It may only be left empty if there is a corresponding value in
the PARVAL1 field
For PARVAL1
- The default value is 50 for all Bdoc types, provided that no entries already exist for individual
Bdoc-types (see PARNAME2)
- Bdoc type-specific entries take priority over general entries (BULK | SUBSET | empty)
- The optimum setting depends on the size of the respective BDoc type (number of segments,
average number of records in dependent segments)
49
- Performance wins can depend on hardware capability, etc.
- Reasonable numeric values for PARVAL1 are relative to the parameters used in the PARNAME2
field as follows:
empty: 50
BULK: 1000 5000
SUBSET: 50
BDOC-Name: 50
- These values should be understood as beginning values. If the values are changed then the
system needs to be monitored to check for timeouts and long runtimes. In these cases the values
should be respectively reduced to acceptable performance levels.
50
Feedback and Questions
Send any feedback by composing an SAP customer message to component SV-GST. You can do
this at http://service.sap.com/message.
51