Академический Документы
Профессиональный Документы
Культура Документы
10
End-to-End Diagnostics
Diagnostics Agent Troubleshooting Guide
Version 3.0, August 2013
END-TO-END DIAGNOSTICS
DIAGNOSTICS AGENT TROUBLESHOOTING
Contents
1 Document Status ................................................................................................................................. 5
1.1 New Document Location .............................................................................................................. 5
1.2 Location of Intermediate Document ............................................................................................ 5
1.3 Old Document Location ................................................................................................................ 5
2 Pre-requisites ...................................................................................................................................... 6
2.1 JDK compatibility .......................................................................................................................... 6
2.2 Managed System ........................................................................................................................... 6
2.3 SLD production system ................................................................................................................ 7
3 Agent installation strategy ................................................................................................................ 10
3.1 Managing system scaling ........................................................................................................... 10
3.2 Managed System environment ................................................................................................... 10
3.3 Managed Systems HA/Failover environment ............................................................................ 10
4 Agent installer.................................................................................................................................... 11
4.1 SAP Host Agent........................................................................................................................... 11
4.1.1 E2E RCA configuration for Solution Manager 7.0x or higher .................................................. 11
4.1.2 E2E RCA configuration for Solution Manager 7.1x only.......................................................... 11
4.2 OS configuration checks ............................................................................................................ 12
4.3 JVM parameters .......................................................................................................................... 12
4.4 Startup scripts ............................................................................................................................. 13
5 Agent connectivity ............................................................................................................................ 17
5.1 Managing system ........................................................................................................................ 17
5.2 Managed system ......................................................................................................................... 19
Connection configuration for managed systems .................................................................................. 22
5.3 Self-Management ........................................................................................................................ 22
5.4 SAP router ................................................................................................................................... 23
6 Agent Maintenance ............................................................................................................................ 24
6.1 Configuration............................................................................................................................... 24
6.1.1 Change Path of the JDK......................................................................................................... 24
6.1.2 Change JVM Parameters ...................................................................................................... 24
6.1.3 Replacing JDKs 1.4.2 with SAP JVM 4 .................................................................................. 24
6.1.4 Verifying Logical hostname .................................................................................................... 25
6.1.5 Usage of smdsetup script ....................................................................................................... 25
6.1.6 Change Agent log level .......................................................................................................... 28
6.2 Managing J2EE user ................................................................................................................... 29
7 Tracking Agent issues ...................................................................................................................... 30
7.1 OSS message pre-requisites ...................................................................................................... 30
7.2 Agent crash investigation........................................................................................................... 30
7.3 Thread dump ............................................................................................................................... 32
7.4 Activate P4 logs .......................................................................................................................... 33
8 Scalability........................................................................................................................................... 34
End-to-End Diagnostics 4
Diagnostic Agent Troubleshooting Guide
©SAP AG 2013
End-to-End Diagnostics 5
Diagnostic Agent Troubleshooting Guide
1 Document Status
Here http://wiki.sdn.sap.com/wiki/display/SMSETUP/Diagnostics+Agent+Troubleshooting
The old location of this document is on the SMP (both this SPM location and the document on SMP are
deprecated/obsolete)
https://service.sap.com/~sapdownload/002007974700000409092009E/DiagAgent_TroubleShooting.pdf
©SAP AG 2013
End-to-End Diagnostics 6
Diagnostic Agent Troubleshooting Guide
2 Pre-requisites
2.1 JDK compatibility
The following table describes the list of different JDKs supported by the Diagnostics Agent for each OS platform:
5.3 64bit on
Note 716927
Power 4, 5
AIX
6.x 64bit on
Note 716927
Power 4, 5
11.31 on IA64
64bit
Note 716926
HP
11.11,
Note 716926
PA-RISC 64bit
SAPJVM5 SAPJVM6 SAPJVM6
10 on SPARC
Solaris Note 716604
64bit / x86_64
IA32 32bit /
Note 716604
IA64 64bit
x86_64 64bit Note 861215
Linux 2 Power 64bit Note 810008
SLES9 or higer
on zSeries Note 746299
64bit
V5R3, V5R4
I5/OS Note 717376
64bit
1.6, 1.7, 1.8, -
Z/OS - -
1.9 64bit
The Diagnostics Agent installation packages 7.1x, 7.20 and 7.30 include the SAPJVM and the SAP Kernel DVD.
Therefore, it is not necessary to install separately the SAPJVM before installing one of those Agents.
The above listed SAP notes could be helpful to determine the adequate JDK version but do not apply JVM
parameters proposed in these SAP notes for the Diagnostics Agent. (They are only intended to SAP J2EE
Engines).
2
Read SAP note 171356 before to install the agent
©SAP AG 2013
End-to-End Diagnostics 7
Diagnostic Agent Troubleshooting Guide
IMPORTANT: In any case refer to the SAP note 1365123 and its attached document.
©SAP AG 2013
End-to-End Diagnostics 8
Diagnostic Agent Troubleshooting Guide
How to check if the Diagnostics Agent is known in a given SLD in order to connect it to Solution Manager?
Go to the SLD with URL http://sldhostname:port/sld
Logon with J2EE user with role SAP_SLD_ADMINISTRATOR
Go to the Administration section
For SLDs based on NW 7.10 and higher, click on the link ‘Maintenance’ / ‘CIM Instances’
For SLDs based on earlier NW releases, click on the link ‘Content Maintenance’
Select the subset: ‘All with Instances’ and Class: ‘SAP_DiagnosticsAgentInstance’
N.B.: No Diagnostics Agent is currently registered, in case the class ‘Diagnostics Agent Instance’ does not
appear in drop down list.
In column “Name” of table, check if the desired Diagnostics Agent is present.
Or since Solution Manager 7.1 use rather the step “Connect Diagnostics Agent” in transaction solman_setup,
scenario “System Preparation”
This step displays for the selected SLD the list of known Diagnostics Agents
Remind that in case the landscape includes several SLDs, having an SLD bridge forwarding
in place, the above approach will ONLY allow validating that a specific Diagnostics Agent is
registered in one of those SLDs. But it does NOT necessarily mean that this Agent is directly
registered in the precise SLD, where the look-up is been performed, like explained above.
This means, to be able to connect Diagnostics Agents using the “Connect Diagnostics
Agents” feature in solman_setup transaction / System Preparation or in Managed System
Configuration / Assign Diagnostics Agent, it is important to select the SLD in which the
Diagnostics Agent is directly registered. This direct registration target SLD is decided and
configured in most cases at Diagnostics Agent installation time.
N.B.: Remind to have a look at the runtime.properties file of the Diagnostics Agent, in order
to find out in which SLD the Agent is directly registered in.
Which J2EE Group is assigned to the SLD user used by the Diagnostics Agent?
The J2EE SAP_SLD_DATA_SUPPLIER group or the security role DataSupplierLD should be assigned to the
user used by the Diagnostics Agent.
What is the frequency on which the Diagnostics Agent checks within its “associated” SLD for Solution
Manager connectivity data updates?
If the Diagnostics agent is registered in an SLD and not yet connected to a Solution Manager System, it will
check every 20 minutes if an association has been created in that SLD, using the “Connect Diagnostics
Agents” feature in solman_setup transaction / System Preparation or in Managed System Configuration /
Assign Diagnostics Agent.
If the Diagnostics agent registered in an SLD and currently connected to a Solution Manager System, it is
going to check every hour if a new / modified association has been created in that SLD.
©SAP AG 2013
End-to-End Diagnostics 9
Diagnostic Agent Troubleshooting Guide
Why is the Diagnostics Agent not visible in the Agent Administration UI?
Verify the below listed items, in case the Diagnostics Agent is not connecting to the Solution Manager system
after you used the “Connect Diagnostics Agents” feature, via solman_setup transaction / scenario: “Managed
System Configuration” / Step: “Assign Diagnostics Agent”, or scenario: “System Preparation”.
N.B.:
Make sure to also search the Agent entry in the Agent Administration UI, using the filter function on the “Host
name” column (visible when the “More details” button is enabled).
Also, neither use the “Server name” (as it might be not assigned yet), nor the “Fully Qualified Hostname” (as
hostname resolution issues will for example make the UI display the physical host) as filter criteria.
You can access the Agent Administration UI in general via a hyperlink displayed in the above mentioned
solman_setup steps.
o Has the adequate SLD been selected when performing the “Connect Diagnostics Agents” operation? See
also above Q&A item.
o Did you wait long enough? See also above Q&A item.
o Is the Diagnostics Agent still able to interact with its “associated” SLD?
Check for SLD CIM or SLD credentials errors in the Diagnostics Agent SMDSystem.?.log log files
o Is the Diagnostics Agent detecting a Solution Manager connectivity data change in the SLD?
Check for other kind of issues in the Diagnostics Agent SMDSystem.?.log log files.
Alternatively simply connect the Diagnostics Agent to the Solution Manager system with the script
‘smdsetup’ (see part 6.1.5 Usage of smdsetup script) instead of using indirectly the SLD to do so.
Note that in case the following message appears in former Agent Candidates UI :
This error simply reports that Solution Manager instance is not found in SLD with the mentioned
hostname BUT it does not prevent the agent attachment process to succeed. The UI will ask for
the hostname and port of Solution Manager during the attachment sequence.
©SAP AG 2013
End-to-End Diagnostics 10
Diagnostic Agent Troubleshooting Guide
The following check list must be validated before installing a new Diagnostics agent on a managed host:
Check whether a Diagnostics Agent is already installed: Verify if the managed system has been installed
with a SAPInst version greater than NW04S SR3 (see part 2.2 Managed System) or check on the file system
whether a folder like /usr/sap/DAA or /usr/sap/SMD already exists. In both cases an agent is already installed
on the system.
If the Diagnostics Agent(s) is/are already installed on the Managed system host(s), check if the
Agent(s) is/are present in Agent Administration UI.
If not, double check whether the Agent(s) is/are known in the SLD where the managed system is
registered. In this last case connect the Agent to the Solution Manager System using the “Connect
Diagnostics Agents” feature (or the Agent candidate Management with Solution Manager 7.0 EHP1).
Alternatively simply connect the Diagnostics Agent to the Solution Manager system with the script
‘smdsetup’ (see part 6.1.5 Usage of smdsetup script) instead of using indirectly the SLD to do so.
In case no Diagnostics Agent is installed, first find out the logical hostname or physical/virtual hostname of the
managed system.
Install the Diagnostics Agent following the rules defined in the document of SAP Note 1365123, on
With Solution Manager 7.1 SP05 and higher refer to the following article to put in place the so called Agents On-the-fly:
Diagnostics+Agent+and+HA+support.
In the context of earlier Solution Manager Releases, it is recommended to install the Diagnostics Agent on
shared/cluster file system. The goal is to move the Agent each time together with the Managed System (only for
Application Server Instances, not ASCS/SCS), but adding the “Agent stop, file system un-mounting, file system re-
mounting and re-start” in the cluster fail-over scripts.
In case the Managed instance and Agent are installed on different file system partitions, it is important when the
Managed SAP instance and the Diagnostics Agent are moved to another cluster node to mount the Agent
Instance file partition before the SAP Instance partition, to allow the Introscope Byte Code Adapter to start.
This is due to the fact that the Wily Setup wizard makes the SAP java instance classpath depending on some jar files
contained in the Diagnostics agent folder, especially if the Introscope Byte Code Adapter has been setup with Solution
Manager (prior to Solution Manager 7.1 SP05). See also Diagnostics+Agent+and+HA+support for further details.
©SAP AG 2013
End-to-End Diagnostics 11
Diagnostic Agent Troubleshooting Guide
4 Agent installer
Therefore add or update the property “service/admin_users” in the SAP Host Agent profile "C:\Program
Files\SAP\hostctrl\exe\host_profile" or “/usr/sap/hostctrl/exe/ host_profile” to list all Diagnostics Agent OS users that
are running on that Physical or Virtual host:
For UNIX:
service/admin_users= <SIDADM USER> <SIDADM USER>
ex:
service/admin_users= daaadm dabadm
©SAP AG 2013
End-to-End Diagnostics 12
Diagnostic Agent Troubleshooting Guide
If the user’s limit parameters for the OS Diagnostics Agent user are not correct, some OutOfMemory exceptions can
for example occur during the extraction activities performed by the Diagnostics Agent.
Also check the note 1437105. Once the limits are modified also remember to restart the Diagnostics Agent
system processes (sapstartsrv) so that the changes are taken into account. For AIX system see note 323816.
If during a zip extractor job (datacollect), an access denied error occurs (could be found in the Diagnostics Agent log
‘SMDApplication.?.log’), double check if the folder /usr/sap/<SID_AGT>/JXX/SMDAgent/temp is writable for the
sapsys group.
Furthermore the file system of the Managed system must be readable by the Diagnostics agent.
Logon with an agent OS user (su – smdadm), go in the managed system folder (ex: /usr/sap/SID/SYS/profile),
and open one profile with the command ‘more <SID>_<INSTANCE_ID>_<HOST>’. If the “more” command is not
working, the access rights are not correct for the Diagnostics agents. In that case, check with your system
Administrator to adapt the file system rights.
©SAP AG 2013
End-to-End Diagnostics 13
Diagnostic Agent Troubleshooting Guide
With agent 7.1x or higher an SAP JVM is used to execute the Diagnostics Agent, the recommended JVM
parameters are:
JDK JVM Parameters
SAP JVM4/5/6 32 bits -DP4ClassLoad=P4Connection -Xmx128m -Xms128m -XX:MaxPermSize=64m
SAP JVM4/5/6 64 bits -DP4ClassLoad=P4Connection –Xmx256m –Xms256m -XX:MaxPermSize=128m
Don’t put the java parameter verbose:gc in the Diagnostics Agents JVM parameters, because this
parameter will steadily increase the size of the file work/std_SMDAgent.out
With Diagnostics Agents 7.11, 7.20 & 7.30 the startup system is compliant to the SAP startup mechanism. It means
that the former commands smdstart.sh and smdstop.sh are no longer available.
Under Windows the SAP MMC allows to start or stop the Diagnostics Agent. The Windows service of the
Diagnostics Agent is configured to start automatically during the Windows startup.
The green status in SAP MMC only indicates that the Diagnostics Agent process is running. It does however
NOT mean that the Agent is operational / connected to the Solution Manager system. Only the Solution
Manager Agent Administration UI allows to confirm that the Diagnostics Agent is successfully connected to
the Solution Manager system and operational. (In the status is gray see 7.2 Agent crash investigation)
Under UNIX, while being logged with the relevant Diagnostics Agent OS User, use the following commands to
operate the Diagnostics Agent.
Command Action
startsap SMDA<Instance_Number> [-v <logical hostname>] Start the Diagnostics agent.
stopsap SMDA<Instance_Number> [-v <logical hostname>] Stop the Diagnostics agent.
The sapstartsrv process of Diagnostics Agent is automatically started during the boot sequence of system if the sapinit
program has been installed on the system. The Diagnostics Agents could later be also stopped and started remotely
with the help of the SAP MC or SAP MMC.
On Windows platforms the SAP MMC allows to start or stop the Diagnostics Agent. The Windows service of
the Diagnostics Agent is configured to start automatically during Windows startup.
The green status in SAP MMC only indicates that the Diagnostics Agent process is running. It does however
NOT mean that the Agent is operational / connected to the Solution Manager system. Only the Solution
Manager Agent Administration UI allows to confirm that the Diagnostics Agent is successfully connected to
the Solution Manager system and operational. (In the status is gray see 7.2 Agent crash investigation)
On UNIX platform, the commands smdstart.sh, smdstop.sh and smdadmin.sh allow to operate the Diagnostics
Agent. Since 7.0 release of the Diagnostics Agent SAPInst, those commands are included in the PATH
©SAP AG 2013
End-to-End Diagnostics 14
Diagnostic Agent Troubleshooting Guide
variable of the OS agent user. In that case, the commands can be called from every folder (not only in the
Agent script subfolder).
Command Action
smdstart.sh <SID> <InstanceNumber> Starts the Diagnostics agent and the sapstartsrv process.
smdstop.sh <SID> <InstanceNumber> Stops the Diagnostics agent and the sapstartsrv process
Launches jcmon tool to administrate the Diagnostics agent
smdadmin.sh <SID> <instanceNumber>
instance
On HP-UNIX platform, an error (!! ERRROR !! current user is not the owner of this instance )
occurs when the smdstart.sh is executed. Check the solution in sap note 1109366.
©SAP AG 2013
End-to-End Diagnostics 15
Diagnostic Agent Troubleshooting Guide
For UNIX system, the Diagnostics Agents do not start automatically when the system reboots because the Diagnostics
Agent startup scripts are not included in the system startup scripts.
The following command line can be used in the system startup scripts:
The following procedure describes how to insert the agent in the boot sequence for a Linux SUSE 10 system:
#!/bin/sh
### BEGIN INIT INFO
# Provides: diagnostics_agents
# Required-Start: $network
# Required-Stop: $network
# Default-Start: 2 3 5
# Description: start diagnostics agents
### END INIT INFO
case "$1" in
'start')
su -c /usr/sap/SMD/J98/script/smdstart.sh > /dev/null 2>&1 -s /bin/sh - smdadm
;;
'stop')
su -c /usr/sap/SMD/J98/script/smdstop.sh > /dev/null -s /bin/sh - smdadm
;;
'status')
su -c /usr/sap/SMD/J98/script/smdadmin.sh -s /bin/sh - smdadm
;;
*)
echo "Usage: $0 { start | stop | status }"
;;
esac
exit 0
When smdstart.sh and smdstop.sh commands are embedded in the system startup scripts, the exit code of
the smd scripts cannot be retrieved. Further more it is important to redirect the standard error output into the
standard output with a syntax like 2>&1 to avoid that the smdstart.sh and smdstop.sh are blocking the parent
script.
©SAP AG 2013
End-to-End Diagnostics 16
Diagnostic Agent Troubleshooting Guide
©SAP AG 2013
End-to-End Diagnostics 17
Diagnostic Agent Troubleshooting Guide
5 Agent connectivity
It is RECOMMENDED to connect the Diagnostics Agent to Solution Manager Java stacks using the J2EE
Message Server. To establish this connection the hostname and port of the Solution Manager J2EE Message
server (Java SCS instance) have to be found out using for example the SCS instance profile.
A. During the installation of a Diagnostic Agent the SAPJ2EE Message server connection mode has to be
selected.
B. This information is stored in the Agent configuration (file runtime.properties) and the credentials are stored
in the Agent secure store (file secstore.properties).
C. When the Diagnostics Agent starts, the P4 connection framework will retrieve from the SCS Message
Server (on HTT port ex: 81xx) all J2EE entry points that are available.
D. Once the list of J2EE entry points is collected, one P4 entry point is chosen from the list.
E. The Diagnostics Agent will then establish a P4 connection to the selected J2EE entry point with the host
name and the port corresponding to that P4 connection.
©SAP AG 2013
End-to-End Diagnostics 18
Diagnostic Agent Troubleshooting Guide
All connection problems are logged in a file under SMDAgent/log/SMDSystem.X.log. Check this log file to
find the root exception of the issue: Once the error exception is found also locate the root cause of the
exception by looking at the last “caused by” sentence.
Exception: com.sap.engine.services.jndi.persistent.exceptions.NamingException:
Exception while trying to get InitialContext. [Root exception is
com.sap.engine.interfaces.cross.DestinationException: there are no access points for
service: P4 registered on the message server]
No P4 entry point has been reported by the Message Server. This is probably due to the fact that it is not
the appropriate Message Server. It has to be the Solution Manager Java stack and not the ABAP stack
Message Server. Also check the text display of the Message server content using the URL
http://msgserver:81xx/msgserver/text/logon. Verify if a P4 entry point is available (as shown below)
Unknown host
It is a network resolution problem due to the fact that the selected P4 hostname retrieved from the SCS
Message server list cannot be resolved from the Managed system host. Retrieve the list of the P4 entries
in the Message server which looks like:
version 1.0
J2EE933894500
J2EE NCED6018XXXXA.dhcp.ncel.sap.corp 59300 LB=1
J2EES NCED6018XXXXA.dhcp.ncel.sap.corp 59301 LB=1
P4 NCED6018XXXXA.dhcp.ncel.sap.corp 59304 LB=1
P4S NCED6018XXXXA.dhcp.ncel.sap.corp 59306 LB=1
P4HTTP NCED6018XXXXA.dhcp.ncel.sap.corp 59305 LB=1
©SAP AG 2013
End-to-End Diagnostics 19
Diagnostic Agent Troubleshooting Guide
As documented in various notes and guides, the Maintenance mode need to be enabled before deploying
a new LM-SERVICE Release, Support Package or Patch, as described also in SAP note 1483508.
If this operation is not performed the Diagnostics Agent runtime behavior will in most cases be affected.
As a consequence the Diagnostics Agent logs might for example contain errors like:
[SMDJNDILookupAgentSideImplementation] lookup
/smdserverctx/ObjAutoRegistered/SMD_DataEnrichment_Registrator failed
[…]
Object not found in lookup of SMD_DataEnrichement_Registrator
Solution Manager SAPJ2EE P4 dispatcher (SAP does NOT recommend to use this mode)
If the Solution Manager system is installed as a cluster (having Dialog Instances), the direct P4 Agent
connection should not be used as the Diagnostics Agent would else always establish a connection to
the same Solution Manager Java instance corresponding to the selected P4 host/port. Also from a
somehow High Availability point of view, you have to use a connection through Solution Manager J2EE
Message server. In that situation, the connection of the Diagnostics Agent will be each time load-
balanced to the different available Java instances.
©SAP AG 2013
End-to-End Diagnostics 20
Diagnostic Agent Troubleshooting Guide
In this example we see that the P4 port is bound for all hosts where a p4 request can be
emitted (* or 0.0.0.0), if the port is bound like “10.45.192.204:50304” only the p4 requests
emitted from the host 10.45.192.204 will be accepted.
It explains why the Diagnostic Agent must be installed on Logical hostname 10.45.192.204 with
SAPInst. The situation can be fixed by changing the profile property SAPLOCALHOST (see
6.1.4 Verifying Logical hostname).
b. After verifying the port in the OS configuration, check in the J2EE dispatcher configuration that
the property “bindHost” is set to 0.0.0.0. This should be visible with the J2EE configuration tool
of the managed system->Instance_xxx->dispatcher->services->p4. If the value is not
configured to 0.0.0.0 (which accepts all hosts), add the Agent IP or install the Agent using
on one of the listed IP corresponding to logical hostnames running on the SAME
underlying physical or virtual host.
2. Check if the Agent can access the J2EE Message Server (SCS) of the Managed system with the host
and the http port stored in the Agent configuration (see the next section). For example :
Open a shell with the agent OS user (<SID>adm) on the Managed System
Execute the command “msprot -l -mshost <host_j2ee_msg_srv> -msserv 39<SCS>
or 36<SCS> -d J2EE –s”, the values of host and port must be retrieved from the Agent
configuration (see the next section)
The command must return the following content :
SAP Message Server Protocol Program, Version 1.3 (built: Jan 24 2008 01:27:19)
INFO CLUSTER-ID HOST GROUP-ID STATE J2EE-SERVICE-MASK
----------------------------------------------------------------------------------------------
©SAP AG 2013
End-to-End Diagnostics 21
Diagnostic Agent Troubleshooting Guide
©SAP AG 2013
End-to-End Diagnostics 22
Diagnostic Agent Troubleshooting Guide
The following table lists all properties generated and updated by the Managed Setup Wizard or solman_setup with
Solution Manager 7.0 EHP1 and higher. These properties allow the Diagnostics Agent to connect to the Managed
system in order to complete the E2E RCA operations. They are accessible using the Agent Administration UI /
Application configuration.
Those properties are stored by server name (scope) of the Diagnostics Agent.
Connection Type to Port Configuration stored in agelet Property
Managed system
P4 port 5xx04 com.sap.smd.agent.application.global.configuration [SID]/[INB]/sapj2ee/P4/port
http port 5xx00 com.sap.smd.agent.application.global.configuration [SID]/[INB]/sapj2ee/http/port
J2EE Message server Intern 39xx com.sap.smd.agent.application.global.configuration [SID]/sapj2ee/msgserver/port
Port
J2EE Message server Http 81xx com.sap.smd.agent.application.global.configuration [SID]/[INB]/sapj2ee/http/port
port
J2EE Message server host <Hostname> com.sap.smd.agent.application.global.configuration [SID]/sapj2ee/msgserver/host
Connection Type Configuration stored in agelet Property Property user Property password
to Managed client
system
ABAP credentials com.sap.smd.agent.application.global [SID]/its/client [SID]/abap/com/user [SID]/abap/com/pwd
7.0x .configuration
ABAP credentials com.sap.smd.agent.application.global [SID]/abap/client [SID]/abap/com/user [SID]/abap/com/pwd
7.1x .configuration
It is recommended to change those values in the Managed system setup wizard or solman_setup with Solution
Manager 7.0 EHP1 and higher instead of directly editing them in the Agent Admin tool. Those values could however be
changed using the Agent Administration tool in case of troubleshooting/debugging.
5.3 Self-Management
During a Self-management setup of Solution Manager, some special configurations in the different setup steps are
needed due to some restrictions about the local Diagnostics Agent connection sharing.
Exception:
com.sap.smdagent.plugins.connectors.p4.exceptions.P4AuthorizationException: Access is
denied: Please make sure the <user> and <password> parameters are correct.; nested
exception is:
com.sap.engine.services.jndi.persistent.exceptions.NamingException: Exception during
getInitialContext operation.No server is running. [Root exception is
com.sap.engine.services.security.exceptions.BaseLoginException: Exception in creating new
RemoteLoginContext instance.]
at
com.sap.smd.agent.plugin.connectors.p4.P4ConnectionService.getP4Connection(P4ConnectionServ
ice.java:69)
...
at java.lang.Thread.run(Thread.java:838)
Caused by: com.sap.engine.services.jndi.persistent.exceptions.NamingException: Exception
during getInitialContext operation.No server is running. Root exception is
©SAP AG 2013
End-to-End Diagnostics 23
Diagnostic Agent Troubleshooting Guide
It is a problem with a Java server level. Two possibilities to fix the issue: (See also SAP Note 1473530)
1. Apply the fix (see SAP note ) on java server and use at least the LM-SERVICE 7.0 EHP1 SP7 P1
2. Use the workaround describes in SAP note (adapt the property bindHost).
Since Solution Manager 7.0 EHP1 SP21, the Diagnostics Agent configuration property – enabling the self-
management scenario – is added automatically by transaction solman_setup / Managed System Configuration
to avoid such a problem. It is named <SID>/selfmonitoring/enabled (value: true) and visible in the Agent
Administration under “com…global.configuration”.
This feature is supported in the context of Solution Manager 7.10 and higher. Also, the automatic activities “Introscope
Host Adapter” and “Introscope Byte Code Adapter installation” in solman_setup transaction / scenario: Managed
system Configuration will use this route provided at the Diagnostics Agent installation/maintenance level.
In the context of Solution Manager 7.0 EHP1 this type of connectivity is not supported. Therefore please contact the
Diagnostics Agent support team to analyze the infrastructure and evaluate the setup possibilities.
©SAP AG 2013
End-to-End Diagnostics 24
Diagnostic Agent Troubleshooting Guide
6 Agent Maintenance
6.1 Configuration
This section describes all procedures to update Diagnostics Agent configurations that could not be changed/updated
centrally using the Solution Manager system.
As the Diagnostics Agent 7.1x and higher run using their own SAPJVM, it is not recommended to change the
path of the JDK.
Some Partner JDK 1.4.2 implementations are going out of maintenance. In this context SAP proposes to replace those
JDKs with the SAP JVM 4. The manual procedure to be followed concerning the Diagnostics Agents is described in the
©SAP AG 2013
End-to-End Diagnostics 25
Diagnostic Agent Troubleshooting Guide
“SAP JVM Switch Tool” guide available under https://service.sap.com/instguides. Especially, consider the chapter 5.
“Performing the Switch to SAP JVM 4 for the Diagnostics Agent”.
To determine the options supported by the smdsetup script, just execute the command “smdsetup.sh” or
“smdsetup.bat”, without any parameters
If the output proposes some action like “managingconf” (in case the smdsetup version is 7.01 or
higher).
Example of smdsetup 7.01 or higher output:
>> Help Information
----------------------
List of actions:
Script Location
©SAP AG 2013
End-to-End Diagnostics 26
Diagnostic Agent Troubleshooting Guide
In 7.0 SR3 and 7.0x the smdsetup script in this version is located in:
/usr/sap/<SID_AGT>/J<Instance_Number>/script
MANAGINGCONF SYNTAX
To execute this command you need to stop the agent. The command updates the connection settings to the Solution
Manager system:
Optionally, you can add the j2ee user information in the command line to update also the agent secure store:
smdsetup managingconf hostname:” sapms://myhost.domain,corp” port:”8101” user:”SMD_ADMIN”
pwd:”init1234”
SLDCONF SYNTAX
To execute this command you need to stop the agent. The command updates the connection settings to the SLD
system of the agent registration:
Register/connect the Agent to the SLD system (http port of SLD java stack).
smdsetup sldconf hostname:”mySLDhost.domain.corp” port:”50000”
Optionally, you can add the SLD user information in the command line to update the agent secure store:
smdsetup sldconf hostname:”mySLDhost.domain.corp” port:”50000” user:”slddsuser” pwd:”xxxxx”
SECSTOREJEE SYNTAX
To execute this command you need to stop the agent. The command updates the credentials of the connection to the
Solution Manager system:
©SAP AG 2013
End-to-End Diagnostics 27
Diagnostic Agent Troubleshooting Guide
SECSTORESLD SYNTAX
To execute this command you need to stop the agent. The command updates the credentials of the connection to the
SLD system:
CHANGESERVERNAME SYNTAX
To execute this command you need to stop the agent. The command updates the “servername” attribute of the agent:
SUPPORTLOGS SYNTAX
This command retrieves all agent log files and executes some basics tests to check the agent pre-requisites as:
©SAP AG 2013
End-to-End Diagnostics 28
Diagnostic Agent Troubleshooting Guide
In case the Diagnostic Agent is currently connected to the Solution Manager System, the procedure to change the log
level of these log location is:
Go in Agent Administration UI
Select the tab “Agent Log viewer”
Select the relevant Agent in the dropdown list
Configure the expected level.
With Solution Manager 7.1, click on the link “Log configuration” to access the Agent log levels configuration UI.
In general, avoid changing the log levels of all Diagnostics Agents at the same time (using button “Save on All
Agents”).
In case the Diagnostics agent is not connected to the Solution Manager System, the procedure to change the log level
is:
For Filename
SMDSystem.log <log-controller effective-severity="INFO" maximum-severity="ALL" minimum-
severity="DEBUG" name="/SMDlogger/System">
SMDAgentApplication.log <log-controller effective-severity="WARNING" maximum-severity="ALL" minimum-
severity="DEBUG" name="/SMDlogger/AgentApp">
P4.log <log-controller effective-severity="NONE" maximum-severity="ALL" minimum-
severity="ALL" name="com.sap.engine.services.rmi_p4">
©SAP AG 2013
End-to-End Diagnostics 29
Diagnostic Agent Troubleshooting Guide
For Diagnostics Agents that have not been connected so far to a Solution Manager 7.1 SP05 system or
higher, remind that the Agent will stop trying to connect/login on the Solution Manager system in case wrong
user credentials have been provided (or the user is locked).
In that context also be aware that in case the SMD_ADMIN user credentials are changed, “offline”
Diagnostics Agent will not get notified of that change. At the time these Agents will go “online” again they will
lock the SMD_ADMIN user, in case at least 3 or more Agents are in this situation.
Since Solution Manager 7.1 SP05, consider using the “Non-authenticated” Agent list, as described by:
http://wiki.sdn.sap.com/wiki/display/SMSETUP/Non-Authenticated+Diagnostics+Agents
Authentication failed
all authentication problems are logged in file SMDSystem.log.
Exception: at com.sap.engine.core.thread.impl3.SingleThread.run
(SingleThread.java:170)
Caused by:
com.sap.engine.services.security.exceptions.BaseLoginException:
Authentication did not succeed.
at
com.sap.engine.services.security.login.ModulesProcessAction.run
(ModulesProcessAction.java:175)
... 13 more
Before restarting the agent It is important to execute some checks about the J2EE user as follow:
1. Connect to the ABAP transaction SU01, to check if the J2EE user (SMD_ADMIN) is not locked and if the
Role assigned is compliant to SAP recommendations done in the “User Administration guide”.
2. Logon to the Agent administration tool with the J2EE user (SMD_ADMIN) to check that the logon module of
the Java stack works correctly.
3. Call the smdsetup to update only the credentials to be sure the password is up-to-date.
4. Restart the Diagnostic agent.
In dev_smdagent.log, if you find the exit code -11112 it is a authentication problem as described above
[Thr 880] Wed Feb 04 09:28:47 2009
[Thr 880] JLaunchIStartFunc: end of main method (rc = 0)
[Thr 6000] JLaunchIExitJava: exit hook is called (rc = -11112)
[Thr 6000] *************************************************************
*** ERROR => The Java VM terminated with a non-zero exit code.
*** Please see SAP Note 943602 , section 'J2EE Engine exit codes'
*** for additional information and trouble shooting.
**********************************************************************
[Thr 6000] JLaunchCloseProgram: good bye (exitcode = -11112)
©SAP AG 2013
End-to-End Diagnostics 30
Diagnostic Agent Troubleshooting Guide
The Diagnostics agent has been installed with SAPinst 7.01 EHP1, 7.11, 7.20, 7.3x or Solution Manager 7.0
EHP1 SR1 or 7.1.
The OS limit for Unix platform are compliant with the agent OS user (<AgentSID>adm)
JDK vendor and version is supported by the Diagnostics Agent
JVM parameters of Diagnostics Agent are compliant to recommendations
The Diagnostics Agent connection to the Solution Manager system is of type J2EE Message Server connection
The Solution Manager Java dispatcher is correctly sized to avoid Agent disconnections (see part 8.1 Solution
Manager J2EE stack settings.)
In case the check points cannot solve the problem, follow the SAP Note 1076573 to create the OSS message.
Also in case the LM-SERVICE version is at least NW04S SPS17 Patch 3, attach the support logs Zip file
generated by the smdsetup script (see part 6.1.5 Usage of smdsetup script).
If the process jlaunch cannot be start, the error is probably located in SAP profile parameters
The Diagnostics Agent does not use specific SAP kernel functions and this message can be ignored.
The process jlaunch is started, open the file dev_SMDAgent which corresponds to the jlaunch process output :
[Thr 3860] JLaunchISetState: change state from [Initial (0)] to [Initial (0)]
[Thr 3860] JLaunchISetState: change state from [Initial (0)] to [Waiting for start (1)]
In this file, all information about agent java process is stored like JVM loading, process state, crash of VM or
bad JVM parameters. It is important to have a look this file to find out Exit Codes or Error messages.
©SAP AG 2013
End-to-End Diagnostics 31
Diagnostic Agent Troubleshooting Guide
Error: [Thr 1084238144] *** ERROR => DlLoadLib: dlopen()= libjvm.so: cannot open shared
object file: No such file or directory -> DLENOACCESS (0, Success) [dlux_mt.c 314]
[Thr 1084238144] *** ERROR => Can't load VM shared library (libjvm.so) (rc=-2)
[jhvmxx_mt.c 1576]
[Thr 1084238144] *** ERROR => Cannot load DLL for Java VM [jlnchxxi_mt.717]
The SAP Note 940893 - Analysis of errors within the startup of J2EE Engine 6.40
can help to troubleshoot JVM loading problems.
If the JVM is loaded correctly, now check the file std_SMDAgent.out and jvm_SMDAgent.out to find some
errors like OutOfMemory, StackOverflow, Nullpointer … which stop the agent process.
©SAP AG 2013
End-to-End Diagnostics 32
Diagnostic Agent Troubleshooting Guide
In SAP MMC, open the node “AS Java Process Table” of Diagnostics Agent identified by the SID and instance number.
Only one process is displayed in the right part: It is the agent process.
Select the process and right click on it to display the context menu, press the item “Dump stack Trace”.
For 7.0x
Logon in UNIX shell with the OS user of the Diagnostics Agent and execute the command “smdadmin.sh XX” (XX is
the instance number of Diagnostics agent). For old agent installation, it is necessary to execute the command
“../smdadmin.sh” from the folder /usr/sap/<SID_AGT>/J<InstanceNumber>/script.
Once the command is executed, the JCMON menu is displaying. Select the entry “20: Local administration menu”.
Then execute the action “10: Dump stacktrace” on process 0 (This is the process of the SMDAgent).
Logon in UNIX shell with the OS user of the Diagnostics agent and go to exe folder C:\usr\sap\DAA\SMDAXX\exe
example:
/usr/sap/DAA/SMDA97.exe > ./sapcontrol -user smdadm abc123 -nr 98 -function J2EEControlProcess SMDAgent
DumpStackTrace
Once the thread dump is done, the output is written in file /usr/sap/<SID_AGT>/J<InstanceNumber>/work/
std_SMDAgent.out with Sun and HP-UNIX JDK.
©SAP AG 2013
End-to-End Diagnostics 33
Diagnostic Agent Troubleshooting Guide
For IBM JDK, the thread dump output is written in java.core.<timestamp>.txt in folder
usr/sap/<SID_AGT>/J<InstanceNumber>/SMDAgent.To find the absolute path of an IBM thread dump, open the file
std_SMDAgent.out and find the line beginning with “Java Dump written to”
The procedure to enable the P4 logs on the Diagnostics agent is the following:
Once the p4 log is enabled on the Agent, go to the Solution Manager system, or the Managed system to execute the
following instructions:
Once all P4 locations are configured, restart the agent to reproduce the problem and attach the agent logs (use the
command ‘smdsetup supportlogs’) and the current defaultrace.log of the Managing or the Managed system to the OSS
message.
Performance Impact
It is important to restore All log locations changed to their default level because the above described
procedure has a high performance impact on the Diagnostics Agent and therefore indirectly on the Managed
system or the Solution Manager system.
©SAP AG 2013
End-to-End Diagnostics 34
Diagnostic Agent Troubleshooting Guide
8 Scalability
The Java dispatcher has an important role in the Agent connection stability. Therefore, you need to adapt - using the
Solution Manager system ConfigTool - the thread and connection settings, as well as the heap size configuration of the
dispatcher node to avoid some timeout connection issues.
Find below a summary of all parameters which have to be adapted. Also remind that a restart of the Solution Manager
Java stack will be required.
After an installation of Solution Manager, the default value of parameter MaxThreadCount in Java Dispatcher
configuration is set to 70. These settings allow by default to connect a maximum of 40 Diagnostics Agents.
©SAP AG 2013
End-to-End Diagnostics 35
Diagnostic Agent Troubleshooting Guide
Connect with telnet program to SAP J2EE server of Solution Manager (default port 5xx18) and go on server node with
command ‘jump <nodeid>’.
Add the group smdserver to enable the commands (use the command ‘add smdserver’).
Once the group smdserver added,
execute the command ’man –g smdserver’ to have access to all available “smd server” commands.
©SAP AG 2013
End-to-End Diagnostics 36
Diagnostic Agent Troubleshooting Guide
Note 1052122 - Host names in SLD You wonder if a hostname that occurs in the System Landscape Directory
(SLD) is correct. In the SLD, hostnames primarily occur in technical systems
and destinations
Note 979277 - Incorrect registration In computer systems registered by the ABAP data supplier in the SLD,
of comupter systems in the SLD attributes may be missing in certain cases (for example, details of the
operating system).
©SAP AG 2013