XenApp 6 Case Studies and Troubleshooting

Rick Berry, Escalation Engineer Mark Callahan, Escalation Engineer May 24th, 2011

Agenda
Case study for UPM issue on XenApp 6 Case study on XenApp 6 filtered policy issue

Questions and wrap-up

Case study for UPM issue on XenApp 6

Problem Definition
Customer was experiencing hung sessions at logon

Some users could log in, others could not

Symptoms
Black Hole User Profile Manager process still running

Logged in users would eventually be affected

Citrix Confidential - Do Not Distribute

Citrix User Profile Manager

Functional Overview - Logon

XenApp Servers

XenDesktop
Streamed/Delivered Desktops

Local Windows Devices

Profiles stored via File Share

My Settings

File Servers

Profile management Service

Active Directory

Functional Details
GPO\User Configuration\Windows Settings\Folder Redirection\My Documents

File Server

\profiles\UserName\

XenApp Server [User Logon Event Location]

\HKLM\Software\Policies\Citrix\UserProfileManager.

File Server

\\server\UserHome\
My Documents

Profile management Service

Troubleshooting Methodology
Complete System Dump PerfMon

User Profile Manager Logs

Citrix Confidential - Do Not Distribute

Troubleshooting Methodology
Complete System Memory Dump

Examine Kernel memory Examine Winlogon process

Citrix Confidential - Do Not Distribute

Troubleshooting Methodology
Performance Monitor

Performance Monitor monitor User Profile Manager and Winlogon threads

PROBLEM NORMAL

Troubleshooting Methodology
User Profile Manager Logs

[PID];WaitUntilChangeJournalIsProcessed: Waiting to finish change journal processing of partition: C Ah Ha! A suspicious log entry!

NTFS Journaling
Event Initial write operation NTFS file system action
The NTFS file system writes a new USN record with the USN_REASON_DATA_OVERWRITE reason flag set. For more information on possible reason flags, see the USN_RECORD structure. The NTFS file system writes a new USN record with the flag setting USN_REASON_DATA_OVERWRITE | USN_REASON_BASIC_INFO_CHANGE. The NTFS file system does not write a new USN record. Because USN_REASON_DATA_OVERWRITE is already set for the existing record, no changes are made to the record. The NTFS file system writes a new USN record with the flag setting USN_REASON_DATA_OVERWRITE | USN_REASON_BASIC_INFO_CHANGE | USN_REASON_DATA_TRUNCATION. If the user making changes is the only user of the file, the NTFS file system writes a new USN record with the following flag setting: USN_REASON_DATA_OVERWRITE | USN_REASON_BASIC_INFO_CHANGE | USN_REASON_DATA_TRUNCATION | USN_REASON_CLOSE.

Setting of the file time stamp Second write operation

File truncation

Close operation

Troubleshooting Methodology
NTFS change journal was showing an increased size of the identification field.

SCREENSHOT

Resolution
Based on the data learned from the NTFS change journal examination, a code change was made to handle changes to the size of the Update Sequence Number record and a hotfix was developed.

Resources discussed

Resources Citrix Profile Manager

Citrix Profile Manager Edocs Site Citrix Profile Manager Logon Diagram

Citrix Profile Manager Logoff Diagram

CTX119791- Profile Management FAQ

CTX12559- Citrix Profile Manager Upgrade FAQ

CTX124455- How to Capture CDF Startup Traces on UPM 3.0

Resources Citrix Profile Manager

Log Parser for Citrix Profile Management Memory Dump File Not Being Generated on Provisioned Target Microsoft Windows Change Journals

Case study on XenApp 6 filtered policy issue

Problem definition

Customer had a new XenApp 6 farm in place XenApp 6 Citrix policies (both computer and user settings) were being applied via Active Directory Group Policy Objects (GPOs) Some of the Citrix policy settings were filtered for Access Gateway connections and others were filtered by client IP When end users connect to the XenApp 6 server from an Access Gateway site, the filtered policy settings were not applying to the session

XenApp 6 policies overview

XenApp 6 Group-based administration

Manage XenApp servers collectively by grouping servers into worker groups
XenApp Farm

You can assign published applications and Citrix policies to worker groups Servers added to worker groups inherit settings

Published Application: Notepad.exe

Worker Group 1

Citrix Policy: Enable Client Drive Mapping

Worker Group 2

Applying Citrix Policies to Worker groups

Worker Group is a new filter for applying Citrix policies

Automatic configuration of new XenApp servers by placing them in an existing worker group

Citrix policy creation and administration

1. Create policies as Citrix IMA-based policies using Delivery Services Console (Used if AD does not exist or access is limited) 2. Create policies as Active Directorybased policies using Group Policy Management Console (GPMC) Note: All Citrix policy settings are configurable using either administration method

Citrix policies via the Delivery Services Console

Citrix policies added via the DSC are stored in the datastore Two types of policies categorized by computer policies and user policies Can be filtered for granular control or unfiltered to apply to all servers or users Policy settings are stored in the servers registry

Filtered versus unfiltered policies

Filtered policy
Applies to specific group of users or servers Uses a variety of filters (IP, AG, Groups, Client name) Use case: Disable CDM for the Marketing domain group

Unfiltered policy
Applies to all servers or users Used when filters or granular control isnt necessary Use case: Specifying the license server that all farm servers will use

Citrix policy extension

Allows integration of Citrix policies into the Windows GPO engine

Adds a Citrix node in the Group Policy Management Console and Group Policy Object Editor Installed with Delivery Services Console
Must be installed on the same machine where Group Policy Objects are administered Can be installed on a standalone machine used for administrative purposes

Citrix policy settings on the server

Computer policies
Enables or disables server settings that were once under the farm and server properties in previous versions Registry location: 32-bit components: HKLM\Software\Policies\Citrix 64-bit components: HKLM\Software\Wow6432Node\Policies\Citrix

User policies
Enables or disables specific features for user sessions Registry location: 32-bit components: HKLM\Software\Policies\Citrix\<SessionID> 64-bit components: HKLM\Software\Wow6432Node\Policies\Citrix\<SessionID>

GPO processing and precedence

OU Group Policy Objects PRECEDENCE Domain Group Policy Objects PROCESSING

Site Group Policy Objects Citrix Group Policy Objects

Local Policies

Citrix policies general roubleshooting checklist

Identify how the policies are being applied (e.g. Active Directory, DSC, both)? Are the Citrix policy files present on the server? What does the group policy results wizard show? CDF Tracing results (see CTX113199 for modules). Setup and review Citrix policy debugging logs. Are the Citrix policy registry settings in place?

Troubleshooting Methodology
Identify how the policies are being applied (e.g. Active Directory, DSC, both)? Are they pulling down properly?

Troubleshooting methodology for the case

Identify how the policies are being applied (e.g. Active Directory, DSC, both)? Are they pulling down properly?

What does output from Group Policy Results Wizard show? Keep in mind GPMC has to be run from XenApp 6 server.

Troubleshooting Methodology
Identify how the policies are being applied (e.g. Active Directory, DSC, both)? Are they pulling down properly?

What does out from Group Policy Results Wizard show? Keep in mind GPMC has to be run from XenApp 6 server.
Enable Citrix policy debugging (see CTX128413)

Setting these values to 0xFFFFFFFF writes the debug information to a log file: %SYSTEMROOT%\Temp\CitrixCseEngine.log Setting these values to 0x0000FFFF writes the debug information to a debugger such as DebugView NOTE: The same values have to be written to HKLM\SOFTWARE\Wow6432Node\Citrix\GroupPolicy

For more details see CTX128413

Troubleshooting Methodology Debug logs

Reviewing %SYSTEMROOT%\Temp\CitrixCseEngine.log we need to verify the logged in user
User Name = REDGETLAB\rickbeuser1, SID = S-1-5-21-39928223702973014269-1922904879-1172, Session ID = 3 Computer Identity - Name = 60426497M1

Next we search on the display name of our policy so we can get the GUID since the GUID is referenced more in the log
Name={52243C73-ED52-4539-B484-02098F5A88F4}, DisplayName=Test Policies, Link=LDAP://OU=RickBe,DC=REDGETLAB,DC=CTX

Troubleshooting Methodology Debug logs

We know that the Access Gateway filter on the policy was using a wildcard (apply to any Access Gateway site), so for the Access Gateway filter we can search on AGInUse FullArmor.GroupPolicyFramework:And(Citrix.Policy.Templates :AGInUse.isValid, Citrix.Policy.Templates:AGFarm.isValid

Citrix.Policy.Templates:WildcardMatch("*"
Citrix.Policy.Templates:AGTags.value,"*",true

Troubleshooting Methodology Registry review

Our session in question was session 3:
HKLM\SOFTWARE\Policies\Citrix\3\Events

"LastUpdate"="2011-03-27 04:12:12Z

Looking at the Evidence key:

HKLM\SOFTWARE\Policies\Citrix\3\Evidence AGFarm= "AGInUse"=dword:00000000

These are issues!!

Root cause isolation

Reviewing the debug logs and comparing this to the registry entries being made allowed us to narrow down the issue to how the policy filters were being evaluated Through our analysis it was determined that there was an issue with the filter expression logic when the Access Gateway filter was being used

Resolution
The investigation into this issue resulted in code change for the Delivery Services Console which was tested successfully by the customer This code change is currently being packaged into a hotfix for the Delivery Services Console

Resources discussed

Resources Citrix Policy Architecture

CTX125152 - Citrix Group Policy Engine Facts in XenApp 6 CTX127612 - How Policies are Applied when an ICA Session Connects to XenApp 6.0 CTX127611 - How Citrix IMA Policies for XenApp 6.0 Fit in to Microsofts GPO Processing and Precedence Model CTX124241 - Technical Guide for Upgrading/Migrating to XenApp 6

Citrix Blog Site - XenApp 6 Policies Deep Dive

Resources Citrix Policy troubleshooting

CTX128413 - XenApp 6 and XenDesktop 5 Group Policy Tracing

CTX111961 - CDFControl Tool

CTX113199 - IMA Modules to Select When Obtaining a CDF Trace for a Policy Problem

Questions?

Before you leave

Session surveys are available online at www.citrixsummit.com starting Thursday, May 26
Provide your feedback and pick up a complimentary gift at the registration desk

Download presentations starting Friday, June 3, from your My Organizer Tool located in your My Synergy Microsite event account