General Questions:

1. When did the problem start? 2. Does it affect all users in the environment? 3. Is it only a particular group of users? 4. Are the affected users all in one Network segment? 5. What version of Outlook? 2007 and 2010 6. Are these imaged builds of Outlook? - Most of them are using imaged build. 7. Do users have the right to install their own software? 8. Can the issue be reproduced, or what is the user doing when the problem occurs? 9. Are there mailbox size limits? 10. Are affected users accessing multiple mailboxes or calendars through their profile? - 11. What happens when Outlook is launched with the /safe switch? 12. Do these users who usually report rpc dialog box issue use any common desktop search application - ? 13. Outlook mode: Are they using online mode or cached mode? 14. How prolific is this ? /What is the frequency of the problem, i.e. does it happen in the morning when users are logging on, throughout the day, etc.? 15. Do we have any knowledge of what actions the users are taking when the popup occurs, i.e. when sending/receiving mail, switching folders, viewing f/b, creating meeting requests, etc.? 16. How long does the pop-up normally stay visible until it goes away? 17. Are there mailbox size limits in place for ALL Users with no exceptions? 18. Was the server name listed in RPC dialog box, if yes, then what is that name? 19. What outlook add in do you have in outlook on client machine? 20. Do the users connect to Outlook using Terminal Server or Citrix based clients? 21. Is Blackberry deployed in your environment? 22.What is the free/busy publishing interval set on the Outlook client? 23. Are users using the Show in Groups feature of Outlook?

Server-side questions ============== 1. What is the version of the Exchange mailbox server including Service Pack level? What is the version of the operating system including Service Pack level? What is the name of the Exchange Server where affected users mailboxes are located? 2. What is the name of the Exchange Server where the affected users default public folder store is located? 3. Are there mailbox size limits in place for all users with no exemptions? [To reduce the impact on server performance, we recommend users run Outlook in Exchange cached mode if there are no mailbox size limits.

Networking related questions ================== 1. Are the affected clients on the same network as the Exchange server (local) or separated by a WAN (remote)? 2. If you have both local and remote users, are both types of users affected by the issue? 3. Are there any network devices between the Outlook clients and the Exchange server, i.e. routers, firewall, etc.? Identify the each hop. 4. Does your Network include a WAN Accelerator? 5. Are all the affected users in the same building/location? 6. What network card and driver is installed on client workstations?

Software related questions ================= 1. Have there been any recent changes in the environment including servers removed, network changes, group policy changes, software updates, security patches, etc.? [Is yes, can you correlate the change(s) made with the initial occurrence of the pop-up?] 2. Do users have the ability to install their own software? [If yes, and Outlook is running in online mode, users may potentially be running desktop search engines against their mailbox which can impact server performance.

Troubleshooting Steps: 1)Always try to ping the CAS server from client machine: Ping<space>CAS server name Note: If there is CAS ARRAY is configured, then ping CAS ARRAY. If while opening outlook , it is taking too much time. Then try to do bypassing the CAS ARRAY.: Steps: to Bypass NLB and go directly to a CAS server and monitor the issue Point one or more problem Outlook clients directly to a single CAS server rather than the CAS array: 1. Close Outlook and Add a line in the Hosts file ("C:\Windows\System32\drivers\etc\hosts") with the IP address of the CAS then a tab and the FQDN of the CAS array name 2. At the command prompt run ipconfig /flushdns 3. At the command prompt run ping <CAS ARRAY NAME> 4. Pinging the CAS Array name should return the IP address of the CAS you've added to the HOSTS file 2)Check tracing route: Run tracert against the CAS ARRAY name/CAS server (if it is single CAS server) and check how many hops it is showing while reaching from destination machine to CAS ARRAY. Ideally, we should get 1 hop , if there is nothing between CAS and client.

If we are getting more than 1 hop , then ask the client , if there is device placed between client and CAS server. 3)Since it is Exchange 2010, so IPv6 should be enabled. So if it disabled , then enable it through network card properties, host file and registry: open the host file and remove the # against the ::001 then save the file. put a checkmark on IPV6 on NIC card from network card properties + open the below registry path and check for the below registry key , if it is there, then remove it to make the IPv6 enable. HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpip6Paramerters :DisabledComponents=4294967295 4)Check the binding order of network cards: Steps:open the network properties, click on advanced>advanced settings>check all the NIC cards are there or not which are enabled , then bring the primary NIC card at the top of all. 5)Test by running outlook in cached mode on 5 work stations and observe if this issue is as prolific as now? 6) Test by running outlook in safe mode for a day or two on 5 more machines and observe if this issue is as prolific as now ? To run the outlook in safe mode, in RUN>type outlook /safe When we run the outlook in safe mode, it opens without third party add ins, if it runs without any issue, then we need to check if there are third party or unwanted outlook add ins enabled or not. If they are enabled, then ask the client to remove them. 7) Make sure client network card drivers are up to date. Run the EXBPA always, then check if it is giving error or warning for updating network drivers on CAS server, if it is there, then ask the Client to update them. 8) Please ask the customer to disable the 3rd party search service that the users are using on their client machines and then check the behavior. 9)If the users are facing this issue at specific point of time, then ask the client to check if there is any other software/script/service is running during the time of the issue. 10)Ask the client whether the users who are facing this issue are from same database or from different database, if they are from same database , then ask the client to move the database for one of the user or the test user to different database, and then check the behavior by opening outlook. 11) Check the SPN on exchange server:

http/ For Exchange Web Services and the Autodiscover service exchangeMDB/ For RPC Client Access exchangeRFR/ For the Address Book service exchangeAB/ For the Address Book service

Note: Load Balancers require the Alternative Service Account and SPN registered to the Load balancer FQDN instead of the individual server names. Note: SPNs pointing to GCs on Exchange2003/ 2007 servers, where as in exchange 2010 it should point to RPC client access server: exchangeAB/<GlobalCatalogServerName> Run following command to check setspn on exchange server: Setspn l <hostname of exchange server> In exchange 2010, if exchangeAB is pointing to global catlog server , then delete that entry: Unregister the exchangeAB SPNs from the Exchange server. To do this, follow these steps: a. At the command prompt, type the following command, and then press ENTER: setspn -D exchangeAB/ GlobalCatalogServerName GlobalCatalogServerName b. At the command prompt, type the following command, and then press ENTER:

setspn -D exchangeAB/ GlobalCatalogServerName.example.com GlobalCatalogServerName

Register the exchangeAB SPNs with the global catalog server. To do this, follow these steps: a. At the command prompt, type the following command, and then press ENTER: setspn -A exchangeAB/Client Access Server Name Client Access Server Name b. At the command prompt, type the following command, and then press ENTER:

setspn -A exchangeAB/ Client Access Server Name .example.com Client Access Server Name 12) Check for the loop back component, and if it is there then delete it: Go to below path:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa Search for the value called DisableLoopbackCheck If it there, then delete it. 13) Disable the TCP chimney offload : On windows 2008 server, following is the command to check TCP chimney status: netsh int tcp show global. To disable TCP Chimney Offload, follow these steps: Use administrative credentials to open a command prompt. At the command prompt, type the following command, and then press ENTER: netsh int tcp set global chimney=disabled To turn off TCP Chimney on windows 2003 server, follow these steps: 1. Click Start, click Run, type cmd, and then click OK. 2. At the command prompt, type Netsh int ip set chimney DISABLED, and then press ENTER. 14) Verifying Exchange Client Access Ports Verify TCP Ports on Exchange Server are listening using the RPCDump i command. Below is example of what to look for. Note: this is a truncated Output. Search for UUIDs. You will see these twice: once for Outlook Anywhere(RPC/HTTP) and once for regular RPC. Connection-oriented TCP/IP using Microsoft Internet Information Server as HTTP proxy LAB-E2K10-CSHT[6001] [5261574a-4572-206e-b268-6b199213b4e4] :ACCESS_DENIED LAB-E2K10-CSHT[6001] [a4f1db00-ca47-1067-b31f-00dd010662da] :ACCESS_DENIED LAB-E2K10-CSHT[6002] [1544f5e0-613c-11d1-93df-00c04fd7bd09] Microsoft Exchange RFR Interface :ACCESS_DENIED LAB-E2K10-CSHT[6004] [1544f5e0-613c-11d1-93df-00c04fd7bd09] Microsoft

Exchange RFR Interface :ACCESS_DENIED LAB-E2K10-CSHT[6002] [f5cc5a18-4264-101a-8c59-08002b2f8426] Microsoft Exchange NSP Interface :ACCESS_DENIED LAB-E2K10-CSHT[6004] [f5cc5a18-4264-101a-8c59-08002b2f8426] Microsoft Exchange NSP Interface :ACCESS_DENIED Connection-oriented RPC TCP/IP LAB-E2K10-CSHT[39627] [5261574a-4572-206e-b268-6b199213b4e4] :YES LAB-E2K10-CSHT[39627] [a4f1db00-ca47-1067-b31f-00dd010662da] :YES LAB-E2K10-CSHT[63534] [1544f5e0-613c-11d1-93df-00c04fd7bd09] Microsoft Exchange RFR Interface :YES LAB-E2K10-CSHT[63534] [f5cc5a18-4264-101a-8c59-08002b2f8426] Microsoft Exchange NSP Interface :YES

A breakdown of example above: Server: LAB-E2K10-CSHT Port: [39627] UUID: [a4f1db00-ca47-1067-b31f-00dd010662da] Accessible: YES

Resource Kit tools: RPC Dump. If one or more of these ports are not listening. You can use Netstat ano and compare the ports that are listed in RPCDump to the PID that is listed in Netstat. Verify if another service has this port. TCP 0.0.0.0:39627 TCP 0.0.0.0:63534 0.0.0.0:0 0.0.0.0:0 LISTENING LISTENING 2804 MSExchangeRPC 5368 MSExchnageAB

Restarting the Information store will not re-register a stolen port, A restart is required to register TCP ports. 15) Since you now connect directly to the RPC Client Access service on the Client Access server for mailbox access or the RPC Client Access service on the Mailbox server for public folder connections, and that clients for directory access connects to the NSPI endpoint, it also means that you by default need to open the TCP 135 EndPointMapper and the dynamic RPC range TCP

1024-65535 between your internal client network and the Client Access servers or arrays and your Mailbox servers. Luckily you can still configure static port mappings just like it has been possible in versions earlier than Exchange 2010. But you have to do so both on any CAS as well as Mailbox servers that are accessed by Outlook MAPI clients. On the CAS servers, for Mailbox connections, you need to use add a DWORD registry key named TCP/IP Port under: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MSExchangeRpc\ParametersSy stem

Figure : Adding the required DWORD key on CAS server to configure a static port number Set the value to the port number to be assigned. In this article we use port 55000, but you are free to choose whatever port you want to use, just remember it should not conflict with other applications using the port. It is recommend you choose a port within the dynamic RPC ranger (1024-65535). To use a static port for public folder access, you need to do the same on the mailbox servers: First open the registry editor. Then add a DWORD key named TCP/IP Port under: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MSExchangeRPC\ParametersSy stem

Its fine to use the same port that you specified on the CAS server.

Figure : Adding the required DWORD key on Mailbox server to configure a static port number Finally you need to limit the port usage for clients that connect to the NSPI endpoint for directory access. Unlike MAPI access to mailboxes and public folders this is done by modifying a file more specifically the Microsoft.exchange.addressbook.service.exe.config configuration file located in the Exchange Bin folder.

Figure : Microsoft.exchange.addressbook.service.exe.config

Open the file in Notepad and then change the RpcTcpPort value from the default assignment of 0 to the port you want Outlook clients and Exchange to use for the directory access via the NSPI EndPoint. In this article we use port 55001.

Figure : Setting a static port for the NSPI EndPoint When you have done the above changes, you should reboot the Mailbox and Client Access on which you performed the above changes. Verifying static ports are used between Outlook and Exchange 2010

Now that we have rebooted the relevant servers, let us verify that Outlook actually connects to Exchange 2010 using the static RPC ports we just specified. To do so first connect to your mailbox using Outlook, then open a command prompt window and type Netstat na

Figure 12: Verifying that Outlook connect to the static RPC ports on the Exchange 2010 servers (client-side) As you can see in Figure 12 above, our Outlook client connects to 192.168.2.238 (Client Access Server) and 192.168.2.239 (Mailbox Server) using port 55000 (MAPI) and port 55001 (DSAccess). Let us also try to run netstat on the Client Access server. As you can see, the client with IP address 192.168.2.198 connects to both port 55000 (static RPC port) and 55001 (static DSAccess port).

+ Checked for registry keys on client machine - all registry keys were present. The following registry values should be there on client machines under HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Rpc\ClientProtocols (Default) REG_SZ (value not set) ncacn_http REG_SZ Rpcrt4.dll ncacn_ip_tcp REG_SZ Rpcrt4.dll ncacn_np REG_SZ Rpcrt4.dll (This may not be there on Vista machines) ncacn_nb_tcp REG_SZ Rpcrt4.dll ncadg_ip_udp REG_SZ Rpcrt4.dll + Checked that some of the users reporting issues are having more than 5000 items in default folders in outlook and also in subfolder. + Also found that some of the public folders also have more than 5000 items . + Pinged casrray -f -l 1472 and found that the MTU allowed on network is 1472 + Pinged all 4 cas servers to check if this MTU is allowed all the way from client to server - Yes - No black hole router. And also deleted the registry for Disabled loopback component. Then Ran Get-Throttling Policy and checked. Ran get-ThrottlingPolicy |Set-ThrottlingPolicy -CPAPercentTimeInMailboxRPC $null -CPAPercentTimeInCAS $null -CPAMaxConcurrency $null RCAPercentTimeInCAS $null -RCAPercentTimeInMailboxRPC $null RCAMaxConcurrency $null -RCAPercentTimeInAD $null -OWAPercentTimeInCAS $null

-OWAPercentTimeInAD $null -OWAMaxConcurrency $null OWAPercentTimeInMailboxRPC $null -EWSPercentTimeInAD $null EWSPercentTimeInMailboxRPC $null -EWSFindCountLimit $null -EWSMaxConcurrenc $null -EWSFastSearchTimeoutInSeconds $null -EWSMaxSubscriptions $null EWSPercentTimeInCAS $null

Action Plan suggested: =============== 1) Among the list of users who report this issue daily, please confirm if they are accessing multiple mailboxes from outlook ? 4) Check for number of items in outlook parent folder and sub folders, number of items should not exceed 500 items http://support.microsoft.com/default.aspx?scid=kb;EN-US;905803 5) Check for number of items in public folders - Folders holding more than 5000 items, move the items to subfolder of the parent public folder.

+ Checked CPU util and Memory Util on all 4 2010 CAS servers. CPU util was normal but Memory util was almost above 85 % on all CAS servers + Checked process to see if any one particular process was causing memory fragmentation. - No, all processes were equally taking up memory resources. + Noticed that ECP App Pool was taking up more memory resources on one of the 4 CAS servers - Recycled the app pool - only 200 MB of Memory was freed after recycling ECP App Pool + Asked customer to increase RAM on all the 4 CAS servers for now to alleviate the problem - customer agreed + Ran performance troubleshooter across all 4 CAS servers and noticed that few IT dept users with large mailboxes were using up to 40 % of MAPI CPU and also were being reported as Highly Active Users. - We identified the same set of users when working earlier on teh same case by running EXMON - These users had more than 10000 items in each of the default folders and more in subfolders in outlook and were running in online mode. This could be the reason. + Asked customer to capture perfmon data all teh 4 CAS servers - Ran EXPERFWIZ for ~ 30 mins on all of them + Checking perfwiz data as per thresholds in http://technet.microsoft.com/enus/library/ff367877.aspx + Looked in perfmon data RPC Average d Latency is between 3-5 MS across all the 4 CAS servers - It is less than 250 MS RPC Requests were also less across all 4 CAS servers - It is less than 40 However, after running through the perfmon data through PAL, noticed that some of the processes are being suspected for causing memory leak due to high handle count. + Customer meanwhile had the RAM increased on all 4 CAS servers to 16 GB

+ Asked customer to capture perfwiz data to look for any perf issues. + Captured perfmon data for another 30 mins + Meanwhile i stressed on the importance of having all client running in cached mode, as its an MS best practice. + Customer dropped off the call to discuss the same with their management and get approval for that. + Also asked customer to find out if its any better now for end users after increasing the amount of RAM on all CAS servers. + Customer to find this information and upload the perfwiz data for further analysis. Looked into perfwiz data, Information on RPC client access counter was not present in 2 of the logs uploaded by customer Checked the perfwiz data of 3rd file uploaded and noticed that RPC averaged latency is less than 250 MS and RPC requests are also less than 40. + Parsed the perfwiz data using PAL and looked for indications of Memory Leak. ( Please find the PAL reports and original perfwiz data in customer files section ) + Checked perfwiz data which got captured after increasing the RAM to see for any improvement in Memory Leak. + Possible Memory leak indicated in second set of PAL reports as well. However, we cannot completely rely on this as perfwiz was run only for ~ 30 mins + Informed customer that the only thing to be done from exchange perspective is to make sure clients run in cached mode and to make sure users having large number of items should be identified and asked to maintain less than 5000 items/folder in outlook as a best practice of Microsoft. Disable NetDMA: To disable NetDMA, follow these steps: 1. Click Start, click Run, type regedit, and then click OK. 2. Locate the following registry subkey, and then click it: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters 3. Double-click the EnableTCPA registry entry. Note If this registry entry does not exist, right-click Parameters, point to New, click DWORD Value, type EnableTCPA, and then press ENTER. 4. To disable NetDMA, type 0 in the Value data box, and then click OK. 5. If the EnableTCPA registry entry does not exist, enable the NetDMA functionality. Related Link: Information about the TCP Chimney Offload, Receive Side Scaling, and Network Direct Memory Access features in Windows Server 2008 http://support.microsoft.com/kb/951037 Follow the below steps at the NIC level on Exchange Servers On the NIC disable the following settings Or send me the screen shot of the NIC advanced settings and I'll let you know what all things to be disabled. This settings vary depending on the manufacturer. Under Network Adapters, double-click the network adapter that you want. On the Advanced tab, click Enabled or Disabled by selecting correct parameter.

Note Different manufacturers may use different terms to describe TCP Chimney Offload on the Advanced properties page of the network adapter. IPv4 Checksum Offload. TCP Checksum Offload (IPv4). Large Send Offload (IPv4) UDP Checksum Offload (IPv4) Link Speed & Duplex - Match the speed on both the sides (Server NIC speed & Port it is connected on Switch/Router) The effect of TCP Chimney offload on viewing network traffic http://blogs.technet.com/b/networking/archive/2008/11/14/the-effect-of-tcpchimney-offload-on-viewing-network-traffic.aspx Added below registry keys on 2 servers ( One mailbox server does not have any production mailboxes and one CAS server) HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MSExchangeRPC\Parameters System Subkey name: EnablePushNotifications Type: DWORD Value: 1 HKLM\System\CurrentControlSet\Services\MSExchangeRPC\ParametersSystem Type: REG_DWORD Subkey name: Maximum Polling Frequency Value: 5000(this is in milliseconds) removed the KeepAliveTime from DCs and increased the KeepAliveTime on Exchange Server, to a default value of 2 hours. Ran the command: Get-ThrottlingPolicy -Identity "DefaultThrottlingPolicy_34ccaae6-7986-4ff692bc-d02ec5c792f1" | FL RunspaceId b4bfcb4bbb1c IsDefault AnonymousMaxConcurrency AnonymousPercentTimeInAD AnonymousPercentTimeInCAS AnonymousPercentTimeInMailboxRPC EASMaxConcurrency EASPercentTimeInAD EASPercentTimeInCAS EASPercentTimeInMailboxRPC EASMaxDevices EASMaxDeviceDeletesPerMonth EWSMaxConcurrency EWSPercentTimeInAD EWSPercentTimeInCAS EWSPercentTimeInMailboxRPC EWSMaxSubscriptions EWSFastSearchTimeoutInSeconds EWSFindCountLimit IMAPMaxConcurrency IMAPPercentTimeInAD IMAPPercentTimeInCAS IMAPPercentTimeInMailboxRPC OWAMaxConcurrency : 27fcbbe8-1266-4970-85b0: : : : : : : : : : : : : : : : : : : : : : : True 1

10

10 10 50 90 60 5000 60 1000

OWAPercentTimeInAD : 30 OWAPercentTimeInCAS : 150 OWAPercentTimeInMailboxRPC : 150 POPMaxConcurrency : 20 POPPercentTimeInAD : POPPercentTimeInCAS : POPPercentTimeInMailboxRPC : PowerShellMaxConcurrency : 18 PowerShellMaxTenantConcurrency : PowerShellMaxCmdlets : PowerShellMaxCmdletsTimePeriod : ExchangeMaxCmdlets : PowerShellMaxCmdletQueueDepth : PowerShellMaxDestructiveCmdlets : PowerShellMaxDestructiveCmdletsTimePeriod : RCAMaxConcurrency : RCAPercentTimeInAD : RCAPercentTimeInCAS : RCAPercentTimeInMailboxRPC : CPAMaxConcurrency : 20 CPAPercentTimeInCAS : 205 CPAPercentTimeInMailboxRPC : 200 MessageRateLimit : RecipientRateLimit : ForwardeeLimit : CPUStartPercent : 75 AdminDisplayName : ExchangeVersion : 0.10 (14.0.100.0) Name : DefaultThrottlingPolicy_34ccaae67986-4ff6-92bc-d02ec5c792f1 DistinguishedName : CN=DefaultThrottlingPolicy_34ccaae6-7986-4ff6-92bc-d02ec5c792f1,CN=Global S ettings,CN=genexservices,CN=Microsoft Exchange,CN=Services,CN=Configuration ,DC=root,DC=forest Identity : DefaultThrottlingPolicy_34ccaae67986-4ff6-92bc-d02ec5c792f1 Guid : e1d0cf4c-7bfe-4064-ab32d4ddde6a2c34 ObjectCategory : root.forest/Configuration/Schema/ms-Exch-Throttling-Policy ObjectClass : {top, msExchGenericPolicy, msExchThrottlingPolicy} WhenChanged : 8/3/2011 10:23:04 AM WhenCreated : 2/24/2011 9:36:30 AM WhenChangedUTC : 8/3/2011 2:23:04 PM WhenCreatedUTC : 2/24/2011 2:36:30 PM OrganizationId : OriginatingServer : phldc01.genexservices.com IsValid : True Educated cx about the Throttling Plicy Ran the command:

set-Throttlingpolicy DefaultThrottlingPolicy_34ccaae6-7986-4ff6-92bcd02ec5c792f1 -RCAPercentTimeInAD 25 -RCAPercentTimeInCAS 500 RCAPercentTimeInMailboxRPC 500 -CPAMaxConcurrency 50 We ran the command to rollback the changes set-Throttlingpolicy DefaultThrottlingPolicy_34ccaae6-7986-4ff6-92bcd02ec5c792f1 -RCAPercentTimeInAD $null -RCAPercentTimeInCAS $null RCAPercentTimeInMailboxRPC $null -CPAMaxConcurrency 20

Action plan : ================ 1. Collect perflogs during the issue. Run it for minimum 2 hours . To run it use experfwiz(http://blogs.technet.com/b/mikelag/archive/2010/07/09/exchange2007-2010-performance-data-collection-script.aspx) command : .\experfwiz.ps1 -server node name -interval 5 threads duration 02:00:00 -filepath D:\Logs. Run it on both the servers. 2. Upload the Experfwiz logs to the SFT workspace ( password and link has been sent in an E-mail) 5. Find the affected user in a particular database and change the RPCclientaccessserver for that database and point it to one CAS server and check if the issue persists. To do that run the command Set-MailboxDatabase <Mailbox Database Name> -RpcClientAccessServer <ClientAccessServer> 6. Please check if there is any other software/script/service is running during the time of the issue. 8. Please refer to the below article as well: http://support.microsoft.com/kb/2535105