Integration of IBM Infrastructure and SAP LVM v2.0

IBM SAP International Competence Center
IBM SAP Technical Brief
Implementing and Using SAP NetWeaver Landscape Virtualization Management on IBM Power Systems and IBM Flex System POWER Nodes
IBM SAP International Competence Center Walldorf, Germany
Version: 2.0 Status: December 2013
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure Copyright IBM Corporation, 2013
Page 1 of 156
Preface
Edition Notice (December 2013) SAP NetWeaver Landscape Virtualization Management software (SAP LVM) is the successor and extension of the SAP Adaptive Computing Controller. This document is complement to the ISICC Implementation Guide SAPs Adaptive Computing on Power Systems, Version 3.0, August 2011. The paper focuses on the new features and uses-cases provided with SAP LVM in conjunction with IBM Systems Director managed Power Systems. It describes the IBM infrastructure specifics required to implement and operate an on-premise Cloud Solution for SAP landscapes. The paper covers a solution stack tested during a Proof-of-Concept at the ISICC. This includes the following components: SAP Business Suite 7 with IBM DB2 UDB LUW 9.7/10.1 databases IBM PureFlex Systems AIX 6 and AIX 7 operating systems IBM Flex System Manager IBM Tivoli Storage FlashCopy Manager IBM SAN Volume Controller IBM XIV Storage System
At the time of writing, SAP LVM V2.0 SP3 is generally available. Scope and Audience People reading this document should understand core components and pre-requisites of the SAP and Systems Software solution stack in an IBM POWER environment including their installation and handling. They should understand the typical use-cases and their execution and effects on a POWER/AIX based SAP infrastructure. The document is intended for IBM (practitioners from Pre- & Post-Sales and Services, GBS/GTS, ATS, eTS,) and customer personnel involved in the implementation of SAP Landscape Virtualization Management software on IBM Power Systems. Since we do not repeat existing documentation and technical information from both IBM and SAP the readers are expected to have a solid knowledge in AIX, PowerVM virtualization, IBM Systems Director and its plug-ins, IBM Flex System Manager, IBM Tivoli Storage FlashCopy Manager and SAP Basis administration.
Authors Dr. Edmund Haefele, IBM Technical Sales for SAP Dr. Thomas Hebert, IBM Technical Sales for SAP Maik Gasterstaedt, IBM SAP International Competence Center Walter Orb, IBM SAP International Competence Center Matthias Koechl, IBM SAP International Competence Center Arnold Beilmann, IBM Research & Development Christoph Langer, IBM Research & Development
Feedback We are interested in any feedback you have. Please send your comments to isicc@de.ibm.com.
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure Copyright IBM Corporation, 2013 Page 2 of 156
Disclaimer This document is subject to change without notification and will not comprehensively cover the issues encountered in every customer situation. It should be used only in conjunction with the product literature accompanying the products listed above. The information contained in this document has not been submitted to any formal IBM test and is distributed AS IS.
Page 3 of 156
IBM SAP International Competence Center 1 INTRODUCTION ....................................................................................................................................... 9 1.1 1.2 1.3 1.4 1.5 2 BACKGROUND AND BUSINESS CONTEXT ................................................................................................... 9 ENABLING END-TO-END MANAGEMENT THROUGH INTEGRATION ............................................................ 9 SCOPE OF THIS COOKBOOK ...................................................................................................................... 10 SAP NETWEAVER LANDSCAPE VIRTUALIZATION MANAGEMENT SOFTWARE (SAP LVM) .................... 11 SAP LVM VERSION 2.0 UPDATE............................................................................................................. 13
OVERVIEW AND PLANNING ............................................................................................................... 15 2.1 2.2 2.3 2.4 2.5 ARCHITECTURE OVERVIEW ...................................................................................................................... 15 COMBINING THE IBM AND SAP LAYERS ................................................................................................. 19 PLANNING PROJECT RESOURCES AND TIMING ......................................................................................... 25 TESTED CONFIGURATION ......................................................................................................................... 27 SAP LVM ENVIRONMENT ....................................................................................................................... 28
INSTALLATION ...................................................................................................................................... 37 3.1 3.2 3.3 3.4 NAMING CONVENTIONS ........................................................................................................................... 38 INSTALL STORAGE INFRASTRUCTURE ...................................................................................................... 39 INSTALL MANAGEMENT ENVIRONMENT.................................................................................................. 49 INSTALL SYSTEM LANDSCAPE ................................................................................................................. 57
INITIAL LVM CONFIGURATION ....................................................................................................... 78 4.1 4.2 4.3 4.4 4.5 4.6 4.7 4.8 4.9 ENGINE SETTINGS .................................................................................................................................... 78 VIRTUALIZATION MANAGER CONFIGURATION IN SAP LVM .................................................................. 78 STORAGE MANAGER CONFIGURATION .................................................................................................... 80 CONFIGURE SAP LVM POOLS ................................................................................................................ 82 CONFIGURE NETWORKS .......................................................................................................................... 83 DISCOVERY OF HOSTS (LPARS) .............................................................................................................. 84 INSTANCE CONFIGURATION/ DISCOVERY IN SAP LVM .......................................................................... 86 ADDITIONAL CONFIGURATION STEPS ...................................................................................................... 93 IBM FLEX SYSTEM MANAGER CONFIGURATION ..................................................................................... 94
DETAILED DESCRIPTION / EXECUTION OF USE CASES ........................................................... 99 5.1 5.2 LVM STANDARD EDITION USE CASES .................................................................................................... 99 SAP LVM ENTERPRISE EDITION USE CASES ........................................................................................ 109
TROUBLESHOOTING HINTS AND TRICKS ................................................................................... 151 6.1 6.2 MANUAL CLEANUP OF SAP LVM COPY PROCESS .................................................................................. 151 DELETE THE SYSTEM IN SAP LVM ....................................................................................................... 153
RESOURCES ........................................................................................................................................... 154 7.1 7.2 IBM DOCUMENTATION ......................................................................................................................... 154 SAP DOCUMENTATION AND SAP NOTES .............................................................................................. 154
Page 4 of 156
List of figures
Figure 1 Logical connections of IBM and SAP components ................................................... 10 Figure 2 Generic SAP NetWeaver LVM capabilities and LVM 2.0 updates (green) .............. 11 Figure 3 SAP LVM Block Diagram (Source: SAP) ................................................................ 16 Figure 4 IBM Systems Director and Plug-Ins .......................................................................... 18 Figure 5 Software structure to integrate IBM IT layers ........................................................... 20 Figure 6 Generic SAP System Copy Workflow ....................................................................... 21 Figure 7 Architecture overview diagram ................................................................................. 28 Figure 8 Redundant virtual fibre channel attachment .............................................................. 33 Figure 9 System configuration in SAP LVM with distributed file system layout ................... 39 Figure 10 Example configuration of an SAP system using GPFS ........................................... 44 Figure 11 Volume Group and Filesystem layout ..................................................................... 46 Figure 12 Volumes of SAP system CL6 .................................................................................. 48 Figure 13 SAP System CL6 in LVM and mount points .......................................................... 49 Figure 14 Overview of the FlashCopy Manager Installation ................................................... 51 Figure 15 Sample LDAP Directory Information Tree for AIX user management .................. 54 Figure 16 Start IPsec ................................................................................................................ 60 Figure 17 Define the Host as Isolation Ready in SAP LVM ................................................ 60 Figure 18 Filesystem for Oracle Redo Logs ............................................................................ 72 Figure 19 LVM Engine Settings .............................................................................................. 78 Figure 20 Virtualization Manager Configuration ..................................................................... 78 Figure 21 Virtualization Manager: Virtualization Manager Types .......................................... 79 Figure 22 Virtualization Manager: Virtualization Manager Properties ................................... 79 Figure 23 Virtualization Manager: Summary .......................................................................... 80 Figure 24 Storage Manager Configuration ............................................................................... 80 Figure 25 Storage Manager: Storage Manager type ................................................................. 80 Figure 26 Storage Manager: Storage Manager Properties ....................................................... 81 Figure 27 Storage Manager: Storage Systems ......................................................................... 82 Figure 28 SAP LVM Pools: Pool Overview ............................................................................ 83 Figure 29 SAP LVM Network Configuration .......................................................................... 83 Figure 30 SAP LVM Network: Basic Configuration ............................................................... 84 Figure 31 SAP LVM Network: Advanced Configuration ....................................................... 84 Figure 32 SAP LVM Network: Summary................................................................................ 84 Figure 33 SAP LVM hosts: Discover Using Host and Instance Agent ................................... 85 Figure 34 SAP LVM Hosts: Add Hosts ................................................................................... 85 Figure 35 SAP LVM Hosts: Edit host properties ..................................................................... 86 Figure 36 SAP LVM Systems: Discover new SAP instances .................................................. 86 Figure 37 SAP LVM Systems: Find Instances/Hosts .............................................................. 87 Figure 38 SAP LVM Systems: Add Instances Modify Host Names .................................... 88 Figure 39 SAP LVM Systems: Add Instances Auto Assign to New or Existing System ..... 88 Figure 40 SAP LVM Systems: Basic Configuration ............................................................... 89 Figure 41 SAP LVM Systems: Provisioning & RFC ............................................................... 89 Figure 42 SAP LVM Systems: Network Isolation ................................................................... 90 Figure 43 SAP LVM Systems: Network Isolation - Add allowed communications ............... 90 Figure 44 Database Configuration: Instance Details ................................................................ 91 Figure 45 Database Configuration: Basic configuration .......................................................... 91 Figure 46 Database Configuration: Instance properties ........................................................... 92 Figure 47 Database Configuration: Mount points .................................................................... 92 Figure 48 Central Services: Mount points................................................................................ 93 Figure 49 Overview of systems and instances ......................................................................... 93
Figure 50 System and AS Provisioning Configuration ............................................................ 94 Figure 51 System Copy Configuration: Release configuration type ........................................ 94 Figure 52 System Copy Configuration: Basic properties......................................................... 94 Figure 53 System Copy Configuration: Installation master ..................................................... 94 Figure 54 IBM Flex System Manager Security: Group membership ...................................... 96 Figure 55 IBM Flex System Manager Security: Create new roles .......................................... 96 Figure 56 IBM Flex System Manager Security: Modify role permissions .............................. 97 Figure 57 IBM Flex System Manager Security: Define static groups ..................................... 98 Figure 58 IBM Flex System Manager Security: Assign groups and roles to user group ......... 98 Figure 59 IBM PowerVM Relocate ....................................................................................... 101 Figure 60 SAP LVM Relocate ............................................................................................... 101 Figure 61 Prepare SAP Instance ............................................................................................. 103 Figure 62 Start SAP system: Mass Operations ...................................................................... 104 Figure 63 Logs of a 'Start SAP system' Operation ................................................................. 104 Figure 64 Relocate SAP System: Mass Operation ................................................................. 105 Figure 65 Shutdown Host ....................................................................................................... 106 Figure 66 SAP LVM Dashboard ............................................................................................ 107 Figure 67 SAP LVM Infrastructure Visualization ................................................................. 108 Figure 68 SAP LVM Performance Monitoring ...................................................................... 109 Figure 69 SAP System Copy Scenarios Supported by SAP LVM (Source SAP) ................. 110 Figure 70 Enable SAP System for SAP System Rename ...................................................... 112 Figure 71 Start SAP System Clone ........................................................................................ 114 Figure 72 SAP System Clone: Basic Data of Target System ................................................. 114 Figure 73 SAP System Clone: Host Selection of Target System ........................................... 115 Figure 74 SAP System Clone: Virtual Host Names and Networks ....................................... 117 Figure 75 SAP System Clone: Storage Volumes ................................................................... 117 Figure 76 SAP System Clone: Database consistency ............................................................ 118 Figure 77 SAP System Clone: Define Allowed Outgoing Connections for System Isolation ................................................................................................................................................ 118 Figure 78 SAP System Clone: Summary ............................................................................... 119 Figure 79 Define RFC Destination for PCA Execution ......................................................... 121 Figure 80 Start SAP System Copy ......................................................................................... 121 Figure 81 SAP System Copy: Basic Data of Target System .................................................. 122 Figure 82 SAP System Copy: Host Selection of Target System ............................................ 122 Figure 83 SAP System Copy: Virtual Host Names and Networks ........................................ 123 Figure 84 SAP System Copy: SAP Instance Numbers .......................................................... 123 Figure 85 SAP System Copy: Storage Volumes .................................................................... 124 Figure 86 SAP System Copy: SVC CLI Warning Message .................................................. 124 Figure 87 SAP System Copy: Database consistency ............................................................. 125 Figure 88 SAP System Copy: User and Group Management ................................................ 125 Figure 89 SAP System Copy: SAPinst Release Configuration .............................................. 126 Figure 90 SAP System Copy: Define Allowed Outgoing Connections for System Isolation 126 Figure 91 SAP System Copy: Post Copy Automation ........................................................... 127 Figure 92 SAP System Copy: Select Task List ...................................................................... 127 Figure 93 SAP System Copy: Summary View ...................................................................... 128 Figure 94 SAP System Refresh: Start System Refresh .......................................................... 129 Figure 95 SAP System Refresh: Basic Data of System to be Refreshed ............................... 129 Figure 96 SAP System Refresh: Host Selection of Target System ........................................ 130 Figure 97 SAP System Refresh: Virtual Host Names and Networks .................................... 130 Figure 98 SAP System Refresh: Storage Volumes ................................................................ 131 Figure 99 SAP System Refresh: Database consistency ......................................................... 131
Figure 100 SAP System Refresh: User and Group Management .......................................... 132 Figure 101 SAP System Refresh: SAPinst Release Configuration ........................................ 132 Figure 102 SAP System Refresh: Define Allowed Outgoing Connection for System Isolation ................................................................................................................................................ 133 Figure 103 SAP System Refresh: Post Copy Automation ..................................................... 133 Figure 104 SAP System Refresh: Storage Volumes .............................................................. 134 Figure 105 SAP System Refresh: Summary .......................................................................... 135 Figure 106 Start SAP System Rename ................................................................................... 136 Figure 107 SAP System Rename: Basic Data of Target System ........................................... 136 Figure 108 SAP System Rename: Virtual Hostnames and Networks .................................... 137 Figure 109 SAP System Rename: SAP Instance numbers ..................................................... 137 Figure 110 SAP System Rename: Mount Data ...................................................................... 138 Figure 111 SAP System Rename: User and Group Management .......................................... 139 Figure 112 SAP System Rename: SAPinst Release Configuration ....................................... 139 Figure 113 SAP System Rename: Define Allowed Outgoing Connections for System Isolation ................................................................................................................................................ 140 Figure 114 SAP System Rename: Post Copy Automation .................................................... 140 Figure 115 SAP System Rename: Summary.......................................................................... 141 Figure 116 Provider Implementation Definition (I) ............................................................... 144 Figure 117 Provider Implementation Definition (II) .............................................................. 144 Figure 118 Provider Implementation Definition (III) ............................................................ 145 Figure 119 Provider Implementation Definition (IV) ............................................................ 145 Figure 120 Custom Hook Definition (I) ................................................................................. 145 Figure 121 Custom Hook Definition (II) ............................................................................... 146 Figure 122 Custom Hook Definition (III) .............................................................................. 146 Figure 123 System State in Operations View ........................................................................ 147 Figure 124 Monitoring View.................................................................................................. 148 Figure 125 Mass Stop and Unprepare Completed ................................................................. 148 Figure 126 Destroy System .................................................................................................... 149 Figure 127 System Destroy: Delete Storage Volumes ........................................................... 149 Figure 128 System Destroy: Host Names .............................................................................. 150 Figure 129 System Destroy: Summary .................................................................................. 150 Figure 130 Remove System in SAP LVM ............................................................................. 153
Page 7 of 156
List of tables
Table 1 Core Functionalities in SAP LVM .............................................................................. 13 Table 2 Supported SAP releases for the SAP LVM scenario .................................................. 15 Table 3 Overview of the Scenarios .......................................................................................... 24 Table 4 Project Phases and estimated time .............................................................................. 27 Table 5 Software Stack ............................................................................................................ 28 Table 6 Operational Model ...................................................................................................... 37 Table 7 Sample directory structure for a distributed file system implementation ................... 39 Table 8 Entries in /etc/services ................................................................................................. 61 Table 9 Local Filesystem layout .............................................................................................. 62 Table 10 XIVGUI Install directory .......................................................................................... 63 Table 11 Minimum filesystem layout (DB2 UDB) .................................................................. 69 Table 12 Extended filesystem layout (DB2 UDB)................................................................... 70 Table 13 Minimum filesystem layout (Oracle) ........................................................................ 70 Table 14 Extended filesystem layout (Oracle) ......................................................................... 71 Table 15 Storage Connection Methods .................................................................................... 81 Table 16 Storage Connection user credentials ......................................................................... 82 Table 17 Required Setup Procedures ..................................................................................... 113 Table 18 Examples for IP addresses ...................................................................................... 115 Table 19 List of SAP Notes.................................................................................................... 155
Page 8 of 156
1 Introduction
1.1 Background and Business Context In the era of worldwide business relations and processes, the flexible and non-disruptive operation of business applications and their underlying middleware and hardware becomes an essential factor for success. Even unforeseen fluctuations in transaction volumes and changing business processes must be reflected immediately by the supporting system capacities. On the other hand, planned interventions such as hardware or application maintenance and testing must not disrupt SAP Business Suite availability to worldwide end users, partners and consumers. In order to accelerate innovation and fast adoption of new business processes, non-production stages of SAP systems need to be deployed on demand, without a long leadtime for procurement and installation. Consequently, manually operating and administrating a growing number of individual systems or landscape components is no longer an option. The concepts of cloud computing promise to save costs and will increase flexibility, elasticity, and automation of system operations to efficiently serve the needs of the business. This paper describes how to design and implement an on-premise cloud environment by combining IBM platform technologies, namely IBM Flex System Manager, IBM Systems Director, and Tivoli Flash Copy Manager with the SAP NetWeaver Landscape Virtualization Management (SAP LVM) environment. Finally, the paper shows how all the use-cases, including cloning, copying, and refreshing of SAP systems, are managed via the SAP LVM administrator user interface. 1.2 Enabling End-to-End Management through Integration
The major advantage of the solution stack described herein is the enablement of end-to-end administration scenarios. This covers both aspects of IT management and SAP basis administration by: 1. Providing a unified administrator interface for SAP topology monitoring and launching automated tasks The SAP basis administrator is familiar with SAP terminologies and user interfaces. Hence, infrastructure complexity and platform specialties are masked and abstracted to that level. Via the SAP LVM Administrator GUI IT tasks are provided to the SAP administrator as self-services. At the same time, the IT administrator keeps control over the infrastructure when enabling the self-services using IBM Systems Director or IBM Flex System Manager. 2. Programmatic integration of IBM and SAP software layers for automation SAP knows best how to manage SAP components and applications. IBM knows best how to manage the infrastructure components consisting of IBM servers, storage, virtualization technology and operating systems. For this reason, IBM platform capabilities and software are used for all infrastructure related tasks. For the same reason, SAP NetWeaver Landscape Virtualization Management complements these for all management and automation functions related to the SAP software stack.
Page 9 of 156
Figure 1 Logical connections of IBM and SAP components The combination of IBM platforms native virtualization and automation capabilities with the SAP management products establishes an excellent foundation for deploying and operating SAP landscapes in an on-premise cloud environment. Consumers of the cloud services are the administrators, not the end-users, though. Line of Business users benefit from faster availability of systems, more flexible testing environments, and demand driven capacity of server and storage resources. 1.3 Scope of this Cookbook This cookbook describes the specifics and installation steps for implementing the SAP NetWeaver Landscape Virtualization Management software V2.0 on an IBM PureFlex System infrastructure. It also provides how-tos for using the resulting administration environment for launching and monitoring the most popular SAP cloud use cases. The installation sequence has been tested in a Proof-of-Concept including SAP Business Suite 7 DB2 UDB LUW IBM POWER AIX 6 and AIX 7 IBM Flex System Manager IBM Tivoli Storage FlashCopy Manager SAN attached disk storage using o IBM System Storage SAN Volume Controller o IBM Storwize V7000 o IBM XIV Storage System Because of the technical affinity of Flex System Manager (FSM) and IBM Systems Director, the content applies to standalone Power systems too. This has been verified in a lab environment and in various customer proof of concept projects.
SAP LVM V1.0 became available in June 2012 and was baseline for the initial version of this document. Since October 2013, SAP LVM version 2.0 is generally available. Our team participated in the SAP LVM 2.0 ramp-up program since April 2013. Therefore, the current document is based on SAP LVM 2.0 already. The fact that we describe a POWER based implementation does in no way mean other hardware and virtualization platforms (including IBM System x) do not offer a comparable degree of SAP LVM integration. The cookbook does not replace a solid technical knowledge of any of the involved layers and is no substitute for existing literature by IBM and SAP. We will point to applicable literature where needed. 1.4 SAP NetWeaver Landscape Virtualization Management software (SAP LVM) SAP NetWeaver Landscape Virtualization Management software allows customers to monitor and manage entire SAP software landscapes and their associated heterogeneous IT infrastructures. SAP LVM provides a single point of control, which allows visualizing, monitoring, and managing data center tasks. The managing aspect includes resource allocation and de-allocation on IT infrastructure level, but also on SAP Business Suite application level.
Figure 2 Generic SAP NetWeaver LVM capabilities and LVM 2.0 updates (green) SAP LVM is an evolution of the previously established SAP Adaptive Computing Controller (ACC 7.3). As such SAP LVM inherits functions like Landscape-wide Visualization and Monitoring and Automated Capacity Management from the former ACC. In addition, SAP LVM provides new and more powerful functions like SAP System Cloning and SAP System Copy / Refresh, which are frequently performed operations at SAP sites. They combine numerous steps on infrastructure and SAP application level
including Post Copy Automation (PCA) sequences. SAP PCA is only available with the SAP LVM Enterprise Edition. SAP LVM Standard Edition is included in a SAP NetWeaver 7.x license without additional fees. The SAP LVM Enterprise Edition is a separately licensed product with additional SAP license fees. SAP LVM provides all the administrative functionalities as listed in Table 1. Functions that require SAP LVM Enterprise Edition are highlighted in blue and italic font. The Standard Edition establishes the basic framework and includes administrative functions that have been available with SAP Adaptive Compute Controller (SAP ACC) in the past. These are related to landscape monitoring and automated startup, stop, and relocate of SAP instances. The Enterprise Edition extends these core capabilities with the advanced functions including/ involving SAP Post Copy Automation sequences. One can mostly automate complex and frequently applied SAP system management tasks. This includes generation of isolated clones of a SAP production system for testing purposes, but also replication of a production system into a QA or pre-production-system that stay fully integrated within the existing transport stages. For latter purpose, business data and system settings need to be replicated, while at all times avoiding any impact on the online production environment.
Page 12 of 156
IBM SAP International Competence Center SAP LVM Functionalities End-to-End SAP System Clone / Copy / Refresh for ABAP End-to-End SAP System Clone / Copy / Refresh for JAVA Post-Copy Automation (PCA) for ABAP and JAVA Automatic Capacity Management (ACM) Application Server Installation / Un-installation Mass Operations (start / stop / relocate) User Configurable Dashboards Reporting Landscape Visualization Custom Operations and Custom Hooks (Extensibility) Custom Services (Management of Non-SAP Components)
Table 1 Core Functionalities in SAP LVM 1.5 SAP LVM Version 2.0 Update SAP LVM version 2.0 introduced a number of new features. Evolutionary enhancements have been made regarding architecture and functionality, and new features had been implemented. The affected areas are highlighted in Figure 2. The most important changes are described in the following three categories: SAP LVM Administrator GUI Changes The user interface allows for more customization. In addition, visualization of SAP landscapes has been improved. Here it is important to note that SAP has changed terminology of managed entities: LVM 1.0 Service Resource Controller Log SAP 2.0 Instance or System Host Log
Customer scripts can extend the default SAP LVM functionality. For these custom operations, new parameters can be defined and applied to managed systems. A search facility allows you to easier identify and navigate to certain instances. System Provisioning Changes In general, LVM 2.0 has introduced some application aware functions for provisioning tasks. E.g., the near-zero downtime (nZDM) cloning scenario for SAP NetWeaver Portal (in SP8) provides a pre-canned procedure for the implementation of support package stacks on a Portal system at minimum business downtime. This minimizes business downtime for all users accessing SAP backend systems through this portal. Further, end-to-end copy and refresh workflows have been extended respectively introduced for SAP ERP/ECC, CRM, SCM and SAP NetWeaver BI. Optionally, these can include the customizable post-copy automation (PCA) phase. PCA can also be launched as standalone sequence on a target system provisioned by other means than SAP LVM.
Page 13 of 156
When executing default LVM sequences enhanced custom hooks increase customization capabilities for platform partners and customers. These hooks are either executed pre-, post, or on-error of operations. Platform Support and Integration Changed Cluster awareness is an interesting point here. However, it does not mean that we now can fully manage PowerHA clusters with LVM 2.0. Currently, this statement means detection and monitoring of multi-node DBs like DB2 LUW DPF or Oracle RAC is supported. So administrators are alerted when manipulating a DB-node. The same applies to shadow DBs, which are mainly used for D/R purposes. For programmatic use and integration into an enterprise monitoring and management system extended outbound interfaces make LVM 2.0 an information provider to any of these 3rd party solutions. In addition, LVM can actively emit notifications about certain events or activities to such solutions. Changes Concerning Supported IBM Storage Solutions With SAP LVM 2.0 the matrix of supported IBM storage solutions was extended. SAP LVM 2.0 supports all IBM storage solutions already supported in SAP LVM 1.0 (IBM XIV, IBM SAN Volume Controller and IBM Storwize V7000) and introduces support for IBM System Storage DS8000. SAP LVM 2.0 improves the execution of SAP system clone, copy, and refresh operations. It provides is more flexibility for specifying characteristics of storage volumes for SAP target systems. The IBM storage solutions now allow to define names for AIX volume groups and logical volumes for the target systems specify the storage pool in the storage system where the target storage volumes will be allocated create target SAP system on remote storage using a pre-configured storage mirroring setup The setup and configuration for SAP system clone, copy, refresh operations with IBM storage solutions was simplified. No additional IBM specific scripts are required anymore. Support for SMI-S communication with storage systems was added. Other communication methods, used in SAP LVM 1.0 before, are now deprecated, but still supported. Changes Concerning Supported IBM Virtualization Solutions With SAP LVM 2.0 the integration with IBM Flex System Manager and IBM Systems Director was improved. Server System Pools in IBM Systems Director VMControl are now supported and may be used as a target for relocation or deployment of logical partitions. For deployment of logical partitions, you may specify multiple network adapters and use IBM Shared Storage Pools as image repository. You may capture logical partitions as image and delete captured images directly from SAP LVM user interface.
Page 14 of 156
2 Overview and Planning

2.1 Architecture overview This section provides an introduction where and how individual functions are implemented. 2.1.1 SAP NetWeaver Landscape Virtualization Management
2.1.1.1 Technical Implementation
SAP LVM is an Java application that runs on add-on to SAP NetWeaver Application Server Java. The required base version for SAP AS Java is 7.30, 7.31 or 7.40. Recommended minimum support packages are SAP NetWeaver Release 7.30 Support Package 08, SAP NetWeaver release 7.31 support package 04 or SAP NetWeaver release 7.4 support package 04. The SAP LVM software is available for any platform that supports those SAP NetWeaver releases. SAP LVM can manage SAP systems running with older releases too. Table 2 lists the supported releases for the managed SAP systems in a SAP LVM scenario. SAP note 1783702 describes in details the restrictions that may apply to some of the older SAP releases.
Supported Managed SAP systems (See SAP Note 1783702) NetWeaver 7.40 NetWeaver 7.30 NetWeaver 7.20 NetWeaver 7.11 NetWeaver 7.10 NetWeaver 7.00 Web AS 6.40 Web AS 6.20 Web AS 6.10 R/3 4.6D R/3 4.6C
Table 2 Supported SAP releases for the SAP LVM scenario Figure 3 illustrates the communication between all the different components. SAP LVM software (both the SAP NetWeaver AS Java and the SAP LVM Add-on) is installed on a single system designated as Management Server. SAP LVM communicates with the managed nodes via the SAP Host Agent (a process running on each managed node) for landscape-wide monitoring and execution of tasks on the systems. Additionally there are direct interfaces (black boxes in the diagram) to the IBM PowerVM virtualization layer and the IBM Storage Layer. The IBM solution described here uses of all those paths.
Page 15 of 156
Figure 3 SAP LVM Block Diagram (Source: SAP) SAP LVM provides a set of APIs as interface. In order to support specific hardware, virtualization, OS and DB layers, each SAP partner needs to supply a set of libraries that translate the generic SAP LVM operations to platform specific commands and executes them as integrated solution on the appropriate component. SAP Note 1783702 describes all the supported platforms for SAP LVM 2.0. Not all platform providers have implemented specific libraries for all of their platforms, or partly they support just a subset of the functionality. Thus, the supported use case for the SAP LVM solution may vary by platform. 2.1.2 IBM Systems Management Components In todays complex and demanding environment, the IT infrastructure needs to react quickly and adapt to the changing needs of the business. More and more of the IT infrastructure is being virtualized and administrators must manage complex interactions between virtualization, compute, storage, and network platforms. System administrators face extraordinary demands as they try to plan, document, and roll out IT infrastructures, identify capacity needs, get the most out of current assets, and contend with constant budget pressure. IBMs Systems Management Components free-up the IT administrators from repetitive, timeconsuming tasks allowing them to pursue higher value projects and accelerate innovation. With more automation and integrated management across infrastructure elements, IT administrators can focus a greater portion of their time on programs that drive innovation and business advantage instead of daily routine tasks. IBM Flex Systems Manager (FSM) and IBM Systems Director are platform management tools. FSM builds upon IBM Systems Director code and its advanced plugins, but both were enhanced with specific elements. They have different specifications and support different types of hardware: Flex Systems Manager (FSM) is a specialized tool for management of IBM Flex Systems and IBM PureFlex Systems IBM Systems Director is a general purpose systems management tool
Page 16 of 156
IBM SAP International Competence Center 2.1.2.1 IBM Systems Management Components for IBM PureFlex Systems
IBM Flex Systems Manager (IBM FSM) is a pre-installed appliance on a specialized compute node for Flex and PureFlex systems. It is optimized for managing the Flex and PureFlex systems. IBM FSM has a deep integration with hypervisors and system patterns for the configuration of the environment on the Flex Chassis. The FSM Explorer user interface (UI) of IBM FSM contains additional functionality and allows a graphical view to the components in the Flex Chassis, the operating system provisioning engine and the configuration patterns for Flex System components. IBM FSM provides a single point of control for management of physical and virtual compute, storage and networking resources from a single management console. IBM FSM enables fast deployment operations of new partitions by simple and repeatable hardware configuration and bare metal provisioning for server nodes, and five click creation and zoning of storage volumes. IBM FSM allows the administrator to monitor the infrastructure from anywhere using iOS, Android and Blackberry mobile devices. IBM FSM is designed to help to get the most out of the IBM PureFlex System or IBM Flex System by automating repetitive tasks, and providing visibility and control across compute, storage, network, and virtualization functions within the data center. IBM FSM management capabilities include: Server management IBM FSM auto-discovers hardware and manages inventory that supports up to 16 managed chassis, 224 compute nodes and 5,000 managed elements. IBM FSM monitors hardwares overall health status to proactively resolve issues. IBM FSM detects and recovers potential problems in the PureFlex environment through an event setup that triggers alerts and actions. Virtualization management (VMControl Express) o Create, edit, manage, and relocates virtual machines for rapid deployment o Discover virtual machines, storage and network resources and visualize the physical-to-virtual relationships Advanced virtualization management (VMControl Standard and Enterprise): o Automate the virtualized environment with system pools Storage management (Storage Control) o Storage provisioning for image creation, deployment, and cloning o Shows relationships between storage and server resources o Policy based storage placement and provisioning. Fabric management (IBM Fabric Manager) o Simplify and manage assignments of Ethernet MAC and Fibre Channel WWN addresses using an enhanced UI o Enables quick and easy deployment, configuration and recovery of compute nodesI o Monitors the health of compute nodes, and automatically without user intervention replaces a failed compute node from a designated pool of spare compute nodes Network management (Network Control) Network resource virtualization and graphical topology view of network resources and
Page 17 of 156
connectivity end to end in a single tool. Energy management (Active Energy Manager) This feature is activated and monitors chassis by default. With simple real-time power and thermal management across compute node, storage, networking and facility providers it can set thresholds on hardware components to drive lower energy costs. Security management Provide administrative capabilities, such as setting up users within profile groups, assigning security level of policies and security governance. FSM can view the access state of managed resources in a web interface display. The integration with SAP LVM leverages the virtualization management of VMControl and Storage Control.
2.1.2.2 IBM Systems Management Components for IBM Power Systems
IBM Systems Director provides similar functionality in an IBM Power Systems environment compared to IBM FSM in an IBM PureFlex System landscape. In contrast to IBM FSM (which is delivered as an appliance), IBM Systems Director must be manually installed on a physical or virtual server. IBM Systems Director is the platform management backbone, providing all the building blocks for integrated services management. It provides the IT administrator with the features: Unified management of physical and virtual resources for IBM servers, storage, and networks. Automated data center operations by implementing cloud-ready virtual infrastructures.
Figure 4 IBM Systems Director and Plug-Ins
Page 18 of 156
Key value of IBM Systems Director is its ability to work in diverse IT environments, so it reduces the number of required management tools and interfaces. IBM Systems Director can manage rack, tower, and blade servers across all IBM brands. IBM Systems Director can manage Flex Systems also, but only in case that no FSM is installed in the Flex Chassis. Optional plug-ins can extend IBM Systems Director. The SAP LVM integration exploits: IBM Systems Director VMControl is a multi-platform virtualization management solution. This solution available as a plug-in for IBM Flex System Manager or for IBM Systems Director provides the following features that are also relevant in any SAP landscape: Create and manage virtual machines Relocate virtual machines Import, edit, create and delete virtual images Deploy new virtual machines from virtual images
The VMControl plug-in is used by SAP LVM to perform the partition and AIX provisioning steps along the overall processing sequence. IBM Systems Director Storage Control is another plug-in for IBM Flex System Manager or for IBM Systems Director. It extends the management of systems to include the storage layer. It is based on IBM Tivoli Storage Productivity and enables functions as: Extends storage management of IBM Systems Director to cover most IBM storage systems Storage device discovery and coverage in integrated physical and logical topology views Show relationships between storage and server resources Ability to configure logical and physical configuration Ability to view controller and volume status and to set notification alerts Integration with IBM Systems Director VMControl storage provisioning for image creation, deployment, and cloning Storage Control is used in the end-to-end provisioning sequence to allow bare-metal LPAR creation and AIX installation: During the provisioning process, Storage Control takes care that the storage for the OS is provisioned and attached to the LPAR.
2.1.2.3 Storage Management (for Snapshots/FlashCopies)
IBM Tivoli Storage FlashCopy Manager. This software provides fast application-aware backups and restores by exploiting snapshot technologies in IBM storage systems. In the context of SAP landscape management, IBM Tivoli FlashCopy Manager is used to create instantaneously consistent copies of online SAP systems. Instead of using these copies as source for backups, SAP LVM uses the flashes as source for further processing system replica by subsequent post-processing steps. 2.2 Combining the IBM and SAP Layers 2.2.1 SAP LVM Interfaces to Management Components
Page 19 of 156
Figure 3 shows how the SAP LVM architecture is implemented for IBM POWER and IBM storage platforms. The integration consists of a set of libraries, which transfer generic SAP LVM commands into the specific syntax for the IBM environment. IBM provides these libraries as part of the SAP LVM software stack. Different communications paths exist between the managing server and the managed nodes: Communication path via the SAP Host Agent The SAP Host agent can execute OS commands on each managed SAP node. SAP LVM communicates with this agent, which calls the vendor libraries in order to map the LVM commands to an OS, DB, or storage specific language and syntax. Communication path using the SAP LVM virtualization adapter The managing system interfaces directly with the IBM Power HMC or with the VMControl plug-in of IBM Flex System Manager/IBM Systems Director to control PowerVM and OS layers. Communication path using the SAP LVM storage adapter (IBM Storage Adapter) For non-disruptive copies of running SAP instances, SAP LVM interacts with the Tivoli Storage Flash Copy Manager, which interfaces to the storage system for the cloning, and handles then handles all storage activities required on the managed nodes also. This implementation guarantees a consistent replication of SAP instances by synchronizing the required steps among the involved infrastructure layers (storage, AIX, DB).
Figure 5 Software structure to integrate IBM IT layers
Page 20 of 156
2.2.2 Example Workflow SAP System Copy In this section, a walk-through through a sample SAP System Copy use case will demonstrate how the integration works together. The workflow is composed of two phases (See Figure 6): System Cloning Phase The system cloning phase begins by creating host name mappings on a name server and is finished when the clones SAP systems is ready to be started in an isolated network segment. These steps are the same for the system copy and system cloning scenarios. At the end of this phase, a 1:1 clone of the source SAP system exists which per default cannot be accessed from other system systems from outside that fenced segment. System Copy Phase After the system-cloning phase is completed, SAP LVM triggers additional automated tasks that make up the system copy phase. This includes renaming the SAP system clone. The Post Copy Automation (PCA-) sequence adjusts the complete SAP configuration so that finally a unique SAP system (SID) is established.
Figure 6 Generic SAP System Copy Workflow We assumes the following landscape example for a detailed description of the workflow: The source system is an SAP application server ABAP system It is deployed in an AIX LPAR The SAP system is configured with virtual IP addresses The operating system, SAP system, and database are installed on separate AIX volume groups The physical disks for these volume groups are allocated on SAN storage systems. The source system is configured in SAP NetWeaver Landscape Virtualization Management
The administrator navigates to the Provision view in the SAP LVM console and selects the source SAP system (All the detailed steps are shown later in chapter 5.2). After pressing the Copy button, SAP LVM guides the administrator through dialogue steps defining the end-toend process. SAP LVM knows all the configuration settings of the registered source system; however, the administrator has to specify configuration parameters for the target system.
Reasonable values for some parameters are determined automatically and pre-set as default during the configuration dialogue. Following parameters can be changed (this is a nonexhaustive list): Target host: SAP NetWeaver Landscape Virtualization Management provides the option to choose an already existing AIX LPAR or to create a new AIX partition as target for the system. (For the latter the virtualization manager configured in SAP NetWeaver Landscape Virtualization Management needs to be set-up for provisioning and configuring such an operating system partition). Host names: The copied SAP system is setup with a new virtual IP address (and IP label) for each of the SAP instances. SAP NetWeaver Landscape Virtualization Management is able to create new addresses and update the domain name server automatically, otherwise the addresses for the target SID need to be pre-configured in DNS. Storage volumes: SAP NetWeaver Landscape Virtualization Management uses the registered storage adapter to gather the data about the volumes attached to the source system. It collects this information at runtime directly from the storage management system. The administrator can specify target volume names and change mount points, if required. Database consistency: SAP NetWeaver Landscape Virtualization Management asks whether the SAP source system is to be stopped during the cloning process or whether the system should stay up and running (for example, in case of a production system). With the online mode, the administrator decides whether SAP NetWeaver Landscape Virtualization Management is responsible for database consistency. Otherwise, there is no guarantee that the cloned database can be recovered and brought online. When selecting database consistency, SAP NetWeaver Landscape Virtualization Management switches the source database into online backup mode (Oracle) or into write suspend I/O mode (DB2). This ensures a consistent database image during the actual cloning step. Target isolation: On the target system network, network fencing has to be established to ensure that the cloned system can be started without interfering with other systems in the data center. This is achieved by using the IPsec feature of AIX. SAP NetWeaver Landscape Virtualization Management allows the definition of permitted outgoing network connections. All other outbound connections are blocked while incoming connections are permitted. SAP copy parameters: SAP NetWeaver Landscape Virtualization Management queries information about the copied SAP system, such as the new SAP system identifier (SAP SID) and master password. SAP copy users: SAP NetWeaver Landscape Virtualization Management allows new users to be created and defined for the copied SAP system on a central user repository.
Page 22 of 156
SAP post copy automation: SAP NetWeaver Landscape Virtualization Management allows the selection of task lists and task list variants to be used for post copy automation. This is possible for multiple clients in the SAP system (such as basis tasks in client 000 and BDLS in client 100).
After all the configuration parameters have been defined, the administrator reviews the input and then starts the SAP system copy workflow. There is a narrated ScreenCam video available on TechDocs, in case you want get a real impression about the look and feel of the SAP LVM administrator console. 2.2.3 Supported Use-Cases and involved components The use-cases for Landscape-wide overview and system monitoring incl. virtualization layers (physical servers and LPARs) Activation, deactivation, and relocation of SAP LVM hosts (LPARs) Provision of new SAP LVM hosts (LPARs) are included in the standard edition of SAP LVM and will leverage one of the virtualization adapters during execution. These use-cases are not dependent on special storage, as long as the storage satisfies the requirements of the virtual environment. IBM Tivoli Storage FlashCopy Manager is not required for execution of these use-cases. Enhanced functionalities (e.g. customized dashboards, generation of reports) will require the enterprise edition. The use-cases for Attaching, detaching, relocating of SAP systems Cloning, copying, refresh of SAP systems have some dependencies on the storage infrastructure. SAP LVM manages the storage volumes belonging to SAP systems during the execution of these operations. The implementation of attach, detach, relocate operations of SAP systems depends on the actual storage type. The functionality for NAS and GPFS is included in the IBM storage library. For SAN-based IBM storage, the IBM Storage Adapter is also required in addition to the IBM storage library. Clone, copy, and refresh operations require SAN based IBM storage, IBM Storage Adapter, and IBM storage and platform libraries. Integration with other, non-IBM storage (like EMC or NetApp) is possible but not covered in this paper. Table 3 summarizes the different use case families in SAP LVM and illustrates which Edition for SAP LVM is required, which SAP LVM adapters needs to be invoked, and which boundary conditions need to be fulfilled for the execution.
Page 23 of 156
Use Case
Landscape Overview System monitoring
VM (LPAR) Activate Deactivate Relocate
VM (LPAR) Provision new LPARs
SAP LVM Adapter
Virtualization Adapter
IBM libraries Not required
Standard Edition SAP LVM Edition

(advanced scenarios require Enterprise Edition)
Standard Edition
Standard Edition
SAP System Attach Detach Relocate Not required for NFS and GPFS, Storage Adapter for SAN based storage IBM storage and platform libraries Standard Edition for NFS and GPFS Enterprise Edition for SAN based storage
SAP System Clone Copy Refresh
Storage Adapter
IBM storage and platform libraries
Enterprise Edition
Storage Management
IBM Power Systems with IBM Systems Director IBM PureSystems with IBM Flex System Manager IBM Power Systems with HMC IBM Tivoli Storage FlashCopy Manager
Virtualization Management
Yes, requires VMControl Express Edition
Yes, requires VMControl Express Edition
Yes, requires VMControl Enterprise Edition and Storage Control
Not required
Not required
Yes
Yes
No
Not required
Not required
Not required
Not required NAS (NFS) GPFS IBM XIV IBM SVC IBM V7000 IBM DS8000
Required IBM XIV IBM SVC IBM V7000 IBM DS8000
Supported Storage
Any
Any
Any
Table 3 Overview of the Scenarios 2.2.4 Custom Cloning Capability and custom hooks/ operations SAP LVM 2.0 introduced a new custom cloning feature. This feature allows creating a system clone based on existing cloning procedures and integrating those into SAP LVM. One can replace the pre-defined sub-steps in the standard workflow of the pre-clone, clone, and post-clone steps by custom defined web services or scripts executed on the source and target systems. The web services/scripts need to be registered as a Provider Implementation Definition and references the script/web services execution triggered by the SAP Host Agent on the source or
Page 24 of 156
target system during the workflow step. The provider implementation definition specifies what is executed on the host or on the SAP instance. When the provider implementation definitions is done, the new custom provisioning process then maps each provider implementations instead of the standard step of the built-in cloning process. It specifies when and under which circumstances the provider implementation definition is executed. Provider implementation definitions can be used in the context of custom hooks or custom operations too: Custom hooks allow to add additional pre-, post-, or error processing steps to the standard workflows defined in SAP LVM. Custom operations allow triggering of user-defined activities directly from the SAP LVM GUI or as part of mass operations. For example, the SAP Basis administrator could run certain custom-defined scripts with privileged permissions on the hosts without granting him root access in general. 2.3 Planning Project Resources and Timing Table 4 summarizes the activities in order to implement a SAP LVM managed POWER landscape comparable to our proof of concept setup described in the following section 2.4. The estimated duration per phase provides a rough guideline only. In detail, the times are dependent on the complexity of the SAP landscape (IT and SAP systems) and its operation as well as on the availability of all the required skills for all involved disciplines.
Page 25 of 156
IBM SAP International Competence Center Project Phase Planning workshop, Verification of landscape prerequisites in client environment Included Tasks / Accomplishments Network and disk design requirements, SW prerequisites Validate that customer has obtained the necessary SAP and AIX software, validate that dedicated IBM hardware infrastructure is available SAP Landscape needs to support virtual host names, and a proper range of IP addresses needs to be available Plan and Prepare Storage LUNs setup of storage Network, SAN zoning infrastructure Eventually map SVC to existing storage Plan and create filesystems Installation and Define LPARs on the managed systems, and configuration steps install AIX OS on the LPARs for the management Set-up virtualization environment environment (IBM o Install and configure IBM Systems Systems Director, Director including VMControl and SAP NetWeaver Storage Control Landscape o Integrate NIM server into System Virtualization Director VMControl environment Management) o Set-up Storage Control for all storage entities (SAN, Storage System) Install and configure storage management software (FlashCopy Manager, Storage CLI) on the management system Plan and configure shared infrastructure services o Install and configure LDAP (if not to be integrated with existing LDAP environment o Plan for DNS Install SAP LVM system Installation and Define AIX golden image content configuration steps Install first AIX LPAR with golden image of managed Install and configure storage management environment incl. software (FlashCopy Manager, SVC CLI) basic testing Install first source SAP system Installation from LVM base configuration scratch for one SAP (Virtualization Manager, Storage Manager System configuration, discover hosts, SAP systems, users) Capture image in Systems Director VMControl for further deployment Additional LVM use case specific configuration (storage, post processing, steps per use case) Note: configuration of PCA post processing requires skilled SAP specialist with both SAP Basis and Application/Business Process knowledge) Installation and Adapt existing SAP system to virtualised configuration steps environment of managed Eventually Upgrade SAP Host Agent environment incl. Install and configure agents and storage basic testing management Migrate/ Customize Install and configure storage management one already existing software (FlashCopy Manager, SVC CLI) SAP system Discovery and setup of LPARs and SAP SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure Copyright IBM Corporation, 2013 Estimated time 2-3 Day workshop with customer
3-4 days working with customer
Two weeks working with customer
Two weeks with customer
At least 2 weeks with customer
Page 26 of 156
IBM SAP International Competence Center Project Phase Included Tasks / Accomplishments systems in IBM Systems Director and SAP LVM Additional LVM use case specific configuration (storage, post processing, steps per use case) Note: configuration of PCA post processing requires skilled SAP specialist with both SAP Basis and Application/Business Process knowledge) Perform functional verification Execute and document each use case o Start / Stop / Relocate of LPARs and SAP systems o Clone SAP Systems (target LPAR available) o Copy SAP Systems (target LPAR available) o Refresh SAP system o Clone SAP Systems (create target LPAR from scratch) o Copy SAP Systems (create target LPAR from scratch) Support customer in LVM acceptance testing from basis or application level Estimated time
Verification Steps using the customers SAP system
At least 2 weeks working with customer
Acceptance Tests
At least 1 week
Document Solution or Handover
at least 2 days for handover, documentation at least 1 week
Table 4 Project Phases and estimated time 2.4 Tested configuration The installation hints in this document are based on a proof of concept installation in the ISICC demo landscape using IBM PureFlex Systems with Power nodes and IBM SAN Volume Controller Storage as well as IBM XIV Storage Systems. Figure 7 illustrates the environment in the proof of concept. In the management environment: Virtualization Manager is IBM Flex System Manager 1.3.0. including VMControl 2.4.3.1 and Storage Management 6.3.3.1. SAP NetWeaver Landscape Virtualization Management software V2.0 SP02 is deployed on top of a SAP NetWeaver Java AS 7.31 engine in a second partition. Additional infrastructure servers like DNS, LDAP server, and NIM server are available and configured.
In the managed environment at least one SAP source system is existing all hosts (AIX LPARs) for both the SAP source system(s) and SAP target system(s) are registered within SAP LVM. The hosts are attached via virtual fibre channel to an IBM XIV Storage system. On each host the software components SAP Host Agent package IBM Tivoli Storage FlashCopy Manager IBM XIV XCLI
are installed and configured.

Management Environment Managed Environment
SAP Source AIX SAP Source AIX SAP Source AIX SAP Source AIX SAP Source AIX SAP Source AIX SAP Component - SAP NetWeaver IBM Tivoli FlashCopy Manager Storage Manager native Interface e.g. IBM XIV XCL VIOS vFC (NPIV)
IBM Flex System Manager VM Control, Storage Manager
SAP LVM AIX SAP NetWeaver Landscape Virtualization Management SAP NetWeaver Java AS IBM Tivoli FlashCopy Manager Storage Manager native Interface e.g. IBM XIV XCLI DNS LDAP IBM Tivoli Directory Server NIM Server AIX
SAP Target AIX
IBM XIV SVC / Storwize

Source
IBM Tivoli FlashCopy Manager Storage Manager native Interface e.g. IBM XIV XCLI
VIOS vFC (NPIV)
Target
Storage connected via vFC (NPIV) Compliant Storage Layout
Figure 7 Architecture overview diagram Table 5 reflects the detailed software stack tested in the scenario:
Software Version
SAP Landscape Virtualization Management Software IBM Flex System Manager / VMControl / Storage Management IBM Script Package SAP Host Agent SAPINST (Rename)1 AIX 7.1 TL2 SP2 IBM Tivoli Storage FlashCopy Manager IBM XIV XIVGUI XCLI DB2 UDB Table 5 Software Stack 2.5 SAP LVM Environment
2.0 SP02 1.3.0 / 2.4.3.1 / 6.3.3.1 7.20 Patchlevel 164 SAP Software Provisioning Manager (SWPM) V1.0 SP4_2 7100-02-02-1316 3.2.0.1 10.2.4.a 3.1 build 18 DB2 v10.1.0.2 (Fixpak 2)
2.5.1 Storage Virtualization A SAP LVM infrastructure requires a centralized storage solution. The storage infrastructure in the ISICC sample environment is based on IBM System Storage solutions using a Storage Area Network (SAN).
Starting with SAP NetWeaver Landscape Virtualization Management 2.0 Support Package 03 the delivery of software component "SAP SYSTEM RENAME 1.0" is part of the "SOFTWARE PROVISIONING MANAGER 1.0" package. SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure Copyright IBM Corporation, 2013 Page 28 of 156
IBM System Storage virtualization products achieve the abstraction from physical volumes of data storage to a logical level. It addresses the increasing complexity of managing storage, while reducing the associated costs. Its main purpose is the full exploitation of the benefits promised by a SAN. Virtualization enables data sharing, ensuring higher availability, providing disaster tolerance, improving performance, allowing for consolidation of resources, providing policy-based automation, and much more besides, which do not automatically result from the implementation of todays SAN hardware components. Storage virtualization is possible on several levels of the storage network components, meaning that it is not limited to the disk subsystem. Virtualization separates the representation of storage to the operating system and its users, from the actual physical components. Storage virtualization accumulates the storage into storage pools, which are independent of the actual layout of the storage (that is, the overall file system structure). Because of this independence, new disk systems can be added to a storage network, and data migrated to them, without causing disruption to applications. Since the storage is no longer controlled by individual servers, it can be used by any server as needed. In addition, it can allow capacity to be added or removed on demand without affecting the application servers. Storage virtualization will simplify storage management, which has been an escalating expense in the traditional SAN environment. IBM SAN Volume Controller (SVC), IBM Storwize V7000, IBM System Storage DS8000, or IBM XIV System Storage is not mandatory for all the use-cases in an adaptive computing environment, but they help in the overall management of the storage landscape. For the SAP System Copy use-cases however the Storage Adapter and Storage Library require one of those systems.
2.5.1.1 SAN Volume Controller System Storage
The IBM (SVC) is an in-band, block-based virtualization product that minimizes the dependency on unique hardware and software, decoupling the storage functions expected in a SAN environment from the storage subsystems and managing storage resources. SVC combines software and hardware into a comprehensive, modular appliance that uses symmetric virtualization. Symmetric virtualization is achieved by creating a pool of managed disks (MDisks) from the attached backend storage systems. Those storage systems are then mapped to a set of volumes for use by attached host systems. System administrators can view and access a common pool of storage on the storage area network (SAN). This functionality helps administrators to use storage resources more efficiently and provides a common base for advanced functions. Key feature exploited for the SAP System Copy scenarios is the ability to create and manage FlashCopies of existing (SAP application) volumes. IBM FlashCopy is a part of the SVC Copy Services and provides a point-in-time copy function. This includes thin-provisioned FlashCopy to make multiple targets affordable.
2.5.1.2 IBM Storwize V7000 Storage System
The IBM Storwize V7000 is a storage server with internal disks, providing the same storage virtualization functions and features like SAN Volume Controller.
2.5.1.3 IBM System Storage DS8000
The IBM System Storage DS8000 is a high-performance, high-capacity, secure storage system designed to deliver the highest levels of performance, flexibility, scalability, resiliency
and total overall value for the most demanding, heterogeneous storage environments. SAP LVM 2.0 supports this IBM storage solution since service pack 3.
2.5.1.4 IBM XIV System Storage
The IBM XIV Storage System is a high-end disk storage series designed to address storage challenges across the application spectrum, including for virtualization, email, database, and analytics and data protection solutions. The IBM XIV Storage System is a fully virtualized system designed to eliminate the need for performance tuning and numerous other storage management activities. IBM XIV provides consistent, hotspot-free enterprise performance and exceptional ease of use. As virtualized storage that meshes tightly with hypervisors, XIV offers optimal agility for cloud and virtualized environments. Key feature exploited for the SAP System Copy scenarios is the ability to create and manage snapshots of existing (SAP application) volumes. The IBM XIV Storage System has taken this concept one step further, offering a completely innovative approach to snapshot creation and management. The XIV system offers clear advantages, including the following:
2.5.1.5
Unlimited number of snapshots in the system. Snapshot creation in virtually zero time, regardless of the size of replicated volumes. Unaffected performance levels in a system that supports snapshots, regardless of the number of snapshots currently defined in the system.
General Parallel File System (GPFS)
Exploiting GPFS is the easiest and most convenient way to implement the classic adaptive scenarios for relocating SAP instances between different hosts. IBM's General Parallel File System (GPFS) provides file system services to parallel and serial applications. GPFS allows parallel applications simultaneous access to the same files, or different files, from any node, which has the GPFS file system mounted, while managing a high level of control over all file system operations. GPFS is particularly appropriate in an environment where the aggregate peak need for data bandwidth exceeds the capability of a distributed file system server. GPFS allows users shared file access within a single GPFS cluster and across multiple GPFS clusters. A GPFS cluster consists of: AIX nodes, a node may be: o An individual operating system image on a single computer within a cluster o A system partition containing an operating system Network shared disks (NSDs) created and maintained by the NSD component of GPFS A shared network for GPFS communications allowing a single network view of the configuration (a single network is used for GPFS communication, including the NSD communication)
All disks utilized by GPFS must first be given a globally accessible NSD name. On AIX server running GPFS, an existing virtual shared disk or physical disk may be given an NSD name.
Page 30 of 156
2.5.2 SAP NetWeaver Landscape Virtualization Management All the base considerations are described in the SAP Notes Note 1783702 - SAP NetWeaver Landscape Virtualization Management 2.0 and Note 1833980 - Integration of IBM Components into SAP LVM 2.0. SAP LVM distinguishes in its configuration between: Instances An instance is a part of an SAP System (database instance, central services instance, application server instance). The SAP LVM use-cases typically are executed on instances or whole SAP systems. Hosts A host describes an operating system installed on a physical or virtual server. SAP instances may be relocated between hosts, or a host may be the target for a new SAP system created via SAP system copy or clone. A new host may be deployed (via SAP LVM and VMControl) so that it can be used as a target for a relocate operation or a SAP system copy, clone, or refresh. SAP LVM monitors hosts through SAP Host Agents running on the operating systems.
SAP LVM Virtualization Management
2.5.2.1
SAP LVM interacts with the virtualization managers to extend host monitoring data available via SAP Host Agents with information about virtualization layers (physical hosts and virtual machines) provision new virtual machines perform load balancing by starting, stopping and relocating virtual machines SAP LVM includes two virtualization managers (adapters) for IBM solutions IBM Flex Systems Manager/ VMControl Adapter IBM Hardware Management Console for IBM Power Both of them enable Monitoring of IBM Power physical servers and logical partitions (LPARs) Management of LPARs (Power on / Power off / Shutdown OS) Relocation of LPARs using Live Partition Mobility Additionally the IBM Flex System Manager/VMControl adapter provides provisioning capabilities for both AIX OS and Storage. See SAP 1833980 - Integration of IBM Components into SAP LVM 2.0 for detailed description of supported configurations and features.
2.5.2.2 SAP LVM Storage Management
SAP LVM interacts with the storage managers (adapters) to clone, copy, or refresh SAP systems prepare, un-prepare, and relocate of SAP instances and systems SAP LVM also includes an IBM Storage Adapter which supports invocation of FlashCopies/SnapShots within the storage system to provide a fast, efficient, and applicationaware cloning of the SAP source system storage volumes.
Page 31 of 156
These functionalities are used in SAP system clone, copy, and refresh operations. In addition, this adapter provides the required functionality to attach and detach storage volumes and filesystems to and from the LPARs during SAP system prepare, unprepared, and relocate activities. IBM Storage Adapter in SAP LVM 2.0 supports following storage system types: IBM XIV Storage System IBM Storwize V7000 storage system IBM SAN Volume Controller (incl. any supported backend storage system)^ IBM System Storage DS8000
See SAP Note 1833980 - Integration of IBM Components into SAP LVM 2.0 for detailed description of supported configurations and features. In this proof of concept we used the following configuration for SAP System Copy use cases: IBM SAN Volume Controller 7.1.0.4 IBM XIV Storage System, Version 10.2 with XIV CLI 3.0
SAN Zoning Requirements
2.5.2.2.1
As a prerequisite for all the SAP System Copy use cases (SAP System clone, copy, refresh) SAN based storage is required for usage with the IBM Storage Adapter. SAN zoning on the target LPAR must be pre-configured. This can be easily fulfilled if the volume(s) for the AIX operating system (rootvg) of the target LPAR is located on the same storage like the SAP systems in scope of the System Copy. Then the existing access to the rootvg on the target LPAR provides SAN access for the SAP application volumes too. In this case, the host and adapter id definitions in the storage system exist already, and the LPAR has an already zoned access in the SAN to the storage system. The SAP application volumes are assigned to the LPAR by LUN masking commands from the SAP LVM storage adapter. The storage system should be zoned according to single initiator zones. Each zone set should contain one (active) virtual fibre channel host port only, but can include multiple storage adapter ports. (e.g. between three and six adapter ports for a storage system). Although not mandatory it is best practise to define the name of the host in the storage system identical to the name of the LPAR or its hostname. If virtual fibre channel adapters are used, then all the WWPNs need to be included in port definition on the storage system, as well as in the SAN zoning definition. This includes all active WWPNs via the VIOS pair, as well as the secondary WWPNs used during a Live Partition Mobility relocation event. 2.5.3 Considerations for the SAP LVM hosts Several storage attachment considerations need to be obeyed, and additional software components need to be installed on the LPARs used as SAP LVM hosts. All these are described briefly in this section.
Page 32 of 156
IBM SAP International Competence Center 2.5.3.1 VIOS / PowerVM and SAN attachment
For all the SAP System clone, copy, and refresh scenarios that invoke storage-based cloning, all the storage volumes for both the source and the target systems need to be attached either via virtual Fibre Channel (vFC) adapters with NPIV or via physical fibre channel adapters attached directly to these systems. In the proof of concept described in this paper we used NPIV attachment via two redundant VIO servers. Each client LPAR is defined with two virtual fibre channel adapters. On each VIO server a virtual fibre channel server adapters is created and assigned to the corresponding client LPAR. Two pairs of WWPNs are created (WWPN1, WWPN2, plus the additional WWPNs for the LPM scenario). We use single initiator zones. Each zone contains one (active) virtual fibre channel host port and between three and six storage system ports. The maximum number of paths that can be zoned to an AIX LPAR is 32. However there is no benefit in having more than 24 paths. Figure 8 illustrates the redundant attachment of LPARs with IBM XIV storage system via two redundant VIO Servers. Each of the VIO servers has one or more physical dual-port fibre channel adapters assigned to the partition. The two ports of the fibre channel adapter card are connected to two different SAN fabrics. Also, the storage system is connected to both fabrics. If an LPAR has only one fibre channel adapter, then zone it to three IBM XIV modules. Zone it to six modules only in case the LPAR has very high throughput requirements.
Fabric #1
Fabric #2
IBM XIV Storage connected via vFC (NPIV)

FC adapter port 0 FC adapter port 1 vFC Server
FC adapter port 0 vFC Server
FC adapter port 1 vFC Server vFC Client WWPN1 vFC Client WWPN2 vFC Client WWPN3 vFC Client WWPN4
vFC Server
VIOS #1
LPAR
VIOS #2
Figure 8 Redundant virtual fibre channel attachment
2.5.3.2
AIX operating system specifics
The AIX operating system on the managed systems needs to fulfil certain criteria: IPsec IPsec is required for network fencing on target hosts for SAP system clone, copy or refresh scenarios and needs to be activated in the OS. Domain Name System For the sake of simplicity, all the virtual hostnames of the SAP instances and systems
Page 33 of 156
should be provided by DNS. One may also configure DNS update functionality in SAP LVM. Local hostname resolution using /etc/hosts must be configured as alternative path. Central User Management is recommended In the test environment, LDAP is used for that purpose. The LDAP filesets need to be installed, and the OS needs to be configured as LDAP client. Services entries We used LDAP as a central repository for the SAP related service entries too. Otherwise an /etc/services file need to be prepared and distributed to all hosts.
IBM Tivoli Storage FlashCopy Manager
2.5.3.3
IBM Tivoli Storage FlashCopy Manager software provides fast application-aware backups and restores, so leveraging advanced snapshot technologies in IBM storage systems. The software provides following features in general: Near-instant application-aware snapshot backups, with minimal performance impact for IBM DB2, Oracle, SAP, Microsoft SQL Server, and Exchange Improvement in application availability and service levels through high-performance, near-instant restore capabilities that reduce downtime Integration with IBM Storwize V7000, IBM System Storage DS8000, IBM System Storage SAN Volume Controller and IBM XIV Storage System on AIX, Solaris, Linux, and Microsoft Windows Fulfilment of advanced data protection and data reduction needs with optional integration with IBM Tivoli Storage Manager
In the context of SAP landscape management, IBM Tivoli FlashCopy Manager is utilized to create instantaneously a consistent copy of an online SAP system.
2.5.3.4 Storage Management Interfaces XCLI
2.5.3.4.1
The IBM XIV Storage System command-line interface (XCLI) provides a mechanism for issuing commands to manage and maintain the XIV storage systems. XCLI commands are entered on the XCLI client either by a user or by another application. The XCLI client and the XIV system communicate using Secure Sockets Layer (SSL) over TCP/IP. . Authentication is based on username/password.
2.5.3.4.1 SAN Volume Controller Command Line Interface (SVC CLI)
The SAN Volume Controller command-line interface (SVC CLI) provides a mechanism for issuing commands to manage and maintain the IBM SVC storage system. Communication is achieved by SSH using SSL over TCP/IP: The SAN Volume Controller cluster acts as the SSH server in this relationship. The SSH client provides a secure environment to connect to the remote machine based on public and private keys for authentication.
2.5.3.4.2 SMI-S
The IBM SVCs CIMOM interface is supplied with the SVC Master Console and is automatically installed as part of the SVC Master Console installation. For the IBM DS8000, the CIMOM interfaces is supplied with the HMC. The CIMOM interface translates an SMI-S
(Storage Management Initiative Specification) command into a proprietary command that the device understands and then convert the proprietary response back into the SMI-S based response.
2.5.3.5 SAP Host Agent
The SAP Host Agent package contains all the required elements for centrally monitoring of any hosts. It is used by SAP LVM to monitor and manage SAP instances and hosts. The SAP Host Agent package needs to be installed and configured on each host containing an SAP component managed by the SAP NetWeaver Landscape Virtualization Management software. The SAP Host Agent package is automatically installed as part of the SAPinst installation procedure for any components of SAP NetWeaver 7.0 EHP2 and higher. However it may need to be updated to a higher patch level to exploit all SAP LVM management features. It is also required to install SAP Host Agent manually on operating systems without any SAP systems. In our case these are operating systems that act as targets in SAP system clone, copy, refresh, or relocate operations. 2.5.4 Considerations for the SAP LVM Instances and Systems All SAP systems need to be installed compliant to a SAP LVM environment. This includes different considerations for user management, network, and storage. Some best practises are discussed in the following.
2.5.4.1 User Management
Ensure that the administrative users are available with same UID and GID on all the LPARs involved in the scenario. A central user management solution provides this requirement and we used LDAP for this purpose.
2.5.4.2 Network Considerations
The following guidelines for the network setup for the SAP system to be managed by SAP LVM needs to be considered: Provide a dedicated virtual hostname for each SAP instance The various instances of an SAP system (SAP database, SAP Central Services, SAP instances) shouldnt share an IP address. Each instance should own a dedicated IP address. Provide a dedicated IP address for each virtual host name Do not use multiple virtual hostnames (IP alias names) referring to one and the same IP address. Ensure that all the virtual host names can be resolved on the SAP LVM system and on all hosts Ensure that the reverse lookup on the SAP LVM server and all other hosts provides a one-to-one mapping
Storage Layout

2.5.4.3
Separate all storage volumes for the different instances During the prepare, unprepared, and relocate activities is must be possible to handle the storage independent of the instances: o Create different AIX volume group for the filesystems related to the SAP Central Instance and the Database
Page 35 of 156
o If a Solution Manager Diagnostics agent exists for the SAP System, place its filesystems to the Central Instance. o If there is a dedicated Solution Manager diagnostics agent for the Database, then its filesystems need to be part of the volume group of the database o If you intend to use FlashCopy backup for the production database also, then split the database content into three VGs: database data (tablespaces), database online logs, and other database files (instance directory, executables, archive logs etc )
Page 36 of 156
3 Installation
The following table provides an overview on the operational model. In the test environment, both the management environment and the managed environment are hosted on IBM AIX partitions. Most of the deployment units for the managed environments were directly added to the AIX operating system image. They were part of the NIM deployment process for a new SAP LVM host and automatically available on new systems.
Node Management Environment Deployment Unit AIX LDAP Client (IBM Tivoli Directory Server Client) SAP NetWeaver 7.3 Java System / DB2 UDB SAP NetWeaver Landscape Virtualization Management 2.0 SP02 IBM Tivoli Storage FlashCopy Manager [acsd] (including GSKit 8 package) XIVGUI XCLI SAP Host Agent package and SAP Adaptive Computing Extensions (SAPACEXT) AIX LDAP Client (IBM Tivoli Directory Server Client) optional IBM Flex System Managers incl. VMControl plug-in IBM Flex System Manager Storage Management (including embedded Storage Control plug-in) AIX, setup as NIM Master LDAP Client (IBM Tivoli Directory Server Client) optional IBM Flex System Manager VMControl subagent for NIM AIX IBM Tivoli Directory Server AIX bind Managed Environment Part of AIX Deployment Unit image template AIX Yes LDAP Client (IBM Tivoli Directory Server Client) Yes SAP Host Agent package and SAP Adaptive Computing Yes Extensions IBM Tivoli Storage FlashCopy Manager [acsgen] Yes (including GSKit 8 package) XIVGUI XCLI Yes (for IBM XIV Storage Systems) SAP System Rename No (required for SAP System Copy on target host) (accessed from a central SAP Software Provisioning Manager NFS share) (required for automated provisioning of SAP Application Server) SAP ECC 6.0 source system DB2 UDB SAP PostCopyAutomation tool (required for SAP System Copy and Refresh)
Management LPAR SAP LVM
Management LPAR IBM Flex System Manager NIM Server LDAP Server DNS Server
Node
on all SAP LVM hosts
on one SAP LVM host
Table 6 Operational Model This chapter describes the installation of the components for both the management and managed environments for an AIX/ POWER environment. Not all the installation steps will be described in all details; it is assumed that the reader has sufficient background knowledge to perform these steps.
Page 37 of 156
In principle the SAP NetWeaver Landscape Virtualization Management server could be deployed on another platform, e.g. Linux/Intel or Microsoft Windows/Intel. Then the deployment unit IBM Tivoli Storage FlashCopy Manager [acsd] would need to be moved to another AIX server (e.g. to the NIM server). XIVGUI XCLI, SAP Host Agent package, and SAPACEXT libraries then would need to be installed on this server too. 3.1 Naming Conventions In a SAP LVM environment SAP systems can potentially run on any SAP LVM enabled server. For example, one could install all SAP system relevant data (database, binaries, and working directories) on a distributed file system. In this case, we recommended putting this data under a common root directory on that distributed file system. A careful planning is required to avoid the collision of various names like IP names, user names, user ids and group ids, directory structures, etc. One method is to include the SAP System ID (SID), which typically is unique in a customer environment, in the required SAP system IP names and root directories of the SAP system data directories. In addition, if you want to run multiple SAP systems or instances on a single compute node, you have to plan for unique SAP instance numbers as well, as you cannot run two SAP instances with the same instance number (but different SIDs) within the same OS image. 3.1.1 Sample Instance IP Names for SID C01 The following is one possible naming convention to build SAP systems IP names. There are no specific requirements for IP names in the SAP LVM infrastructure, however choosing and sticking to one convention will help to simplify the administration. c01dbs c01scs c01pas c01as1 IP name for database server IP name for SAP central services instance IP name for SAP primary application server instance IP name for additional application server instance
3.1.2 Sample directory structure for SID C01 on a Distributed File System Each SAP system should have a common high level subdirectory name to provide a standard entry point to all data that belongs to the same system. For example if the distributed file system is mounted at /sapfs, then all data belonging to SAP system C01 could be allocated under the subdirectory /sapfs/sapC01. The root directory for the SAP system C01 would then be /sapfs/sapC01. In that root directory, all instance relevant subdirectories are stored. The mappings between the physical locations on the storage subsystem and the directory locations as they are expected by the SAP kernel have to be defined in SAP LVM. The controller will then create the required mapping between the new data location and the required standard path before starting an instance on a server. Table 7 shows a sample directory structure for SID C01:
Directory <root>/home/c01adm <root>/db2 <root>/oracle <root>/sapmnt/C01 <root>/usr/sap/c01scs <root>/usr/sap/c01as1 /sapfs/saptrans01 Content Home directory for SAP admin user Database server, client, instance Directory for executables, profiles, shared files Instance directories for central services instance Instance directories for application server 1 Common transport directory Path in standard installation /home/c01adm /db2 /oracle /sapmnt/C01 /usr/sap/C01 /usr/sap/C01 /usr/sap/trans Page 38 of 156
Table 7 Sample directory structure for a distributed file system implementation The screenshot in Figure 9 shows the SAP instance configuration screen in SAP LVM, which illustrates the mapping between physical layout and the required layout for the SAP database instance EC1. The storage type is Distributed File System (DFS) with a subtype of gpfs", which refers to a General Parallel File System installation. The entries with mount option dirmount define the required mapping between the physical location and the mount point that is expected by the SAP kernel.
Figure 9 System configuration in SAP LVM with distributed file system layout 3.1.3 Sample directory structure using Network Attached Storage (NAS) The same concept as in the previous section can be used to layout the physical data locations on a NAS device. A typical organization unit with NAS is called a volume where you store directories and files. Those resources then have to be exported to the clients that need to access them. The root directory for SAP system C01 could for example look like nas_server:/vol/sapvol1/sapC01, where nas_server is the IP name of the NAS device. Again the mappings between physical locations and expected locations have to be defined in the SAP LVM Controller. The controller will then create the required mappings (in this case using client NFS mount instead of creating symbolic links) before starting a instance on a server. 3.2 Install Storage Infrastructure
3.2.1 Exploit GPFS as adaptive storage This section provides a brief summary of setup tasks to get started with a GPFS cluster using the ISICC SAP LVM landscape implementation as an example. Using GPFS it is possible to carry out the use cases of the standard edition of LVM including start, stop, and relocate of SAP systems. The use-cases for System Clone, Copy, and Refresh are currently not supported with GPFS. We do not cover in detail typical GPFS administration tasks like GPFS installation and administration, the creation and management of storage pools, definition of file placement policies, etc. For those tasks you need to refer to the standard product documentation and available Redbooks. Especially to configure data availability and disaster recovery scenario, you should carefully review the GPFS Base documentation (e.g. the GPFS Concepts,
Page 39 of 156
Planning, and Installation Guide and the GPFS Advanced Administration Guide). Additional information can be found in the GPFS Wiki
3.2.1.1 Planning for GPFS Quorum Nodes
3.2.1.1.1
The first task is to decide on a node quorum strategy. As described in the GPFS Concepts, Planning, and Installation Guide, GPFS enforces a node quorum rule to prevent multiple nodes from assuming the role of the file system manager in the event of a network communication problem in the cluster. A majority of quorum nodes must remain active in order for the cluster to sustain normal file system usage (multi-node quorum). The ISICC implementation uses a new node quorum mechanism that was introduced with GPFS V2.3 node quorum with tiebreaker disks, which allows you to run with as little as one quorum node available as long as you have access to a majority of the quorum disks. Please refer to Chapter 2 in the GPFS Concepts, Planning, and Installation Guide for a detailed description of the quorum concept and guidelines to select quorum nodes. In the ISICC setup, we use two quorum nodes and one tiebreaker disk.
3.2.1.2 Install and Setup GPFS Install GPFS Software
3.2.1.2.1
You need to install the following filesets: gpfs.base gpfs.msg.en_US This is best accomplished by including these filesets in a NIM installp bundle.
3.2.1.2.2 Create GPFS Cluster
The mmcrcluster is used to create a GPFS cluster. Our setup uses two quorum nodes: siccps10 and siccps12. The same nodes are also used as primary (option p) and secondary (option s) configuration server to store the GPFS cluster configuration data. For cluster communication we use ssh and scp instead of rsh and rcp, which has to be specified with the r and R options. The A option specifies that GPFS daemons are to be automatically started when nodes come up. The default is not to start daemons automatically. Our cluster is named siccfs.isicc (-C option).
root> mmcrcluster N siccps12:manager-quorum,siccps10:manager-quorum p siccps12 s siccps10 r /usr/bin/ssh R /usr/bin/scp C siccfs.isicc A
After a successful creation of the cluster, you can verify your setup using the mmlscluster command.
root> mmlscluster GPFS cluster information ======================== GPFS cluster name: GPFS cluster id: GPFS UID domain: Remote shell command: Remote file copy command:
siccfs.isicc 691765743288319456 siccfs.isicc /usr/bin/ssh /usr/bin/scp
GPFS cluster configuration servers: ----------------------------------Primary server: siccps12 Secondary server: siccps10
Page 40 of 156

Node Daemon node name IP address Admin node name Designation ---------------------------------------------------------------------------------------------1 siccps12 9.153.165.71 siccps12 quorum-manager 2 siccps10 9.153.165.69 siccps10 quorum-manager
Then you need to startup GPFS on all nodes and you should check the state of the cluster:
root> mmstartup -a root> mmgetstate -aL Node number Node name Quorum Nodes up Total nodes GPFS state Remarks -----------------------------------------------------------------------------------1 siccps12 2 2 8 active quorum node 2 siccps10 2 2 8 active quorum node
3.2.1.2.3
Define Network Shared Disk (NSD) Devices
GPFS uses Network Shared Disk (NSD) devices to provide access to the disks. A NSD device can either be physically attached (using a SAN infrastructure) to a node or it can be accessed using a virtual connection (through the network) using a primary and, if specified, backup NSD server. GPFS determines if a node has physical or virtual connectivity to an underlying NSD through a sequence of commands invoked from the GPFS daemon. This determination is called disk discovery and occurs at both initial GPFS startup as well as whenever a file system is mounted. In the ISICC proof of concept landscape, we used the following configuration file to create the NSD devices:
/var/mmfs/conf/nsd.disks: #Description of disk attributes #<disk name>:<primary NSD server>:<2ndary NSD server>:<disk usage>:<failure group>:<NSD name> #Tiebreaker disk hdisk3:::descOnly:-1:nsd_tbrk_01: #Data and metadata disk for /siccfs hdisk4:siccps12:siccps10:dataAndMetadata:1:nsd_siccfs_01: hdisk5:siccps12:siccps10:dataAndMetadata:1:nsd_siccfs_02:
The first disk (hdisk3) holds file descriptor data only and is used as a tiebreaker disk. hdisk4 and hdisk5 are designated as data disks for our LVM cluster file system. We use our quorum nodes as primary and backup NSD servers as well, although all of our nodes do have physical access to the disks. This provides some automatic fallback capability should the SAN access become unavailable. Create the NSD devices using the mmcrnsd command and verify your setup using the mmlsnsd command (the option -v yes specifies that the NSD are to be created only if the disk has not been formatted by a previous invocation of the mmcrnsd command):
root> mmcrnsd -F /var/mmfs/config/nsd.disks -v yes root> mmlsnsd -aL File system Disk name NSD volume ID Primary node Backup node --------------------------------------------------------------------------------------------siccfs nsd_siccfs_01 0999A547447D7EFE siccps12 siccps10 siccfs nsd_siccfs_02 0999A547447D7F00 siccps12 siccps10 (free disk) nsd_tbrk_01 0999A547447D7DA6 (directly attached)
The specified disk file is rewritten during NSD creation. Each hdisk entry is preceded by a comment (# sign) and followed by the equivalent NSD entry. The new file can then be reused for file system creation during subsequent configuration steps. The file system create command uses NSD devices and not AIX hdisks.
IBM SAP International Competence Center 3.2.1.2.4 Activate Node Quorum using Tiebreaker Disks
You need to shutdown the GPFS cluster and use the mmchconfig to activate the tiebreaker disks. Use the following commands to activate the NSD device nsd_tbrk_01 as a tiebreaker disk:
root> mmshutdown a root> mmchconfig tiebreakerDisks="nsd_tbrk_01" root> mmstartup a
You can use the mmlsconfig and mmgetstate commands to check your configuration after the switch to tiebreaker mode:
root> mmlsconfig Configuration data for cluster siccfs.isicc: ------------------------------------------clusterName siccfs.isicc clusterId 691765743288319456 clusterType lc autoload no useDiskLease yes maxFeatureLevelAllowed 903 tiebreakerDisks nsd_tbrk_01 [siccps12] takeOverSdrServ yes File systems in cluster siccfs.isicc: -----------------------------------/dev/siccfs root> mmgetstate -aL Node number Node name Quorum Nodes up Total nodes GPFS state Remarks -----------------------------------------------------------------------------------1 siccps12 1* 2 8 active quorum node 2 siccps10 1* 2 8 active quorum node
In the mmlsconfig output you can see that device nsd_tbrk_01 is used as tiebreaker disk. The quorum information in the mmgetstate output is displayed as 1*, which indicates that this is a two node tiebreaker disk cluster.
3.2.1.2.5 Create GPFS File System
The last step is to create the file system itself. We used the following excerpt of the disk file that was created by the mmcrnsd command as input to create file system command:
/var/mmfs/config/disks.siccfs: #Description of disk attributes #<disk name>:<primary NSD server>:<2ndary NSD server>:<disk usage>:<failure group>:<NSD name> #Data and metadata disk for /siccfs # hdisk4:siccps12:siccps10:dataAndMetadata:1:nsd_siccfs_01: nsd_siccfs_01:::dataAndMetadata:1:: # hdisk5:siccps12:siccps10:dataAndMetadata:1:nsd_siccfs_02: nsd_siccfs_02:::dataAndMetadata:1::
The mmcrfs command was used to create the /siccfs cluster file system:
root> mmcrfs /siccfs /dev/siccfs -F /var/mmfs/config/disks.siccfs A automount -B 256K -v yes
We used the A automount option to specify that the file system is automatically mounted when its first accessed.
3.2.1.2.6 Mount GPFS File System
If you did not use the automount option during file system creation, you can use the standard AIX mount command to mount the cluster filesystem:

root> mount /siccfs
3.2.1.3
Considerations for SAP Instance Installation on GPFS
A SAP system has to be up and running before it can be registered and configured in the Landscape Virtualization Management software. SAP LVM is not able to create the required environment (mount target directories) for a new installation, this has to be done manually (see also section 3.4.5). You should first create the directory structure on the GPFS file system according to your naming conventions and then mount the subdirectories at the mount points that are expected by the SAP installation routines. The following commands create a sample directory structure for system C01:
mkdir mkdir mkdir mkdir mkdir mkdir /siccfs/sapC01 /siccfs/sapC01/oracle /siccfs/sapC01/sapmnt /siccfs/sapC01/usrsap /siccfs/sapC01/trans /siccfs/sapC01/c01adm
Then you need to create the required mapping before you can start the installation. This is done with a normal mount command using the option v mmfs (you may have to create some of the directories for the mount points first, e.g. mkdir /home/c01adm).
mount mount mount mount mount -v -v -v -v -v mmfs mmfs mmfs mmfs mmfs /siccfs/sapC01/oracle /siccfs/sapC01/sapmnt /siccfs/sapC01/usrsap /siccfs/sapC01/trans /siccfs/sapC01/c01adm /oracle /sapmnt/C01 /usr/sap/C01 /usr/sap/trans /home/c01adm
After the installation is finished and the new SAP instance is registered at the Solution Manager, you should unmount all directories and proceed to the SAP LVM configuration tasks.
3.2.1.4 Instance Configuration in SAP LVM for GPFS
When you enable SAP systems in SAP LVM, you have to configure the data storage on the System tab in the Configuration menu (for further details see chapter 4.7). Data residing on a GPFS file system must be defined with Storage Type DFS. The Partner Id is ibm and FS/SRID Type is gpfs. Please note that these values are case sensitive and have to be entered using lower case. The Export Path designates the directories as they are stored in the GPFS file system and the Mount Point/SRID is the mount point as expected by the SAP kernel. The last field is Mount Options, the available options for GPFS file systems are fsmount and dirmount. The first one can be used to mount the GPFS file system itself (in case you dont use the GPFS automount option) and the second option to specify the subsequent subdirectory mounts. For example to mount subdirectory /usr/sap/EC1 on a GPFS file system that does not use the automount option, you would need the following two entries: Storage Type Mount Point/SRID Export Path Mount Options FS/SRID Type DFS /siccfs /dev/siccfs fsmount gpfs
Page 43 of 156
Partner ID Storage Type Mount Point/SRID Export Path Mount Options FS/SRID Type Partner ID
ibm DFS /usr/sap/EC1 /siccfs/sapEC1/usrsap dirmount gpfs ibm
Figure 10 shows an example configuration of a GPFS based SAP system within the ISICC SAP LVM landscape. The database used in this example is Oracle. For DB2 based systems, the filesystems need to be created and mounted accordingly to the DB2 file system structure. Note that a separate pool for SAP systems using the GPFS file systems was created in SAP LVM.
Figure 10 Example configuration of an SAP system using GPFS

3.2.1.5 Add a Node to GPFS Cluster
You need to add an AIX node to the GPFS cluster before you can mount the cluster file system. This is done with the mmaddnode command after the GPFS software installation is completed:
root> mmaddnode N siccps18
You can remove a node from the cluster using the mmdelnode command and we recommend that you remove the node before you start a NIM re-installation of a specific node. You can list the state of cluster nodes with the mmgetstate command:
root> mmgetstate -aL Node number Node name Quorum Nodes up Total nodes GPFS state Remarks -----------------------------------------------------------------------------------1 siccps12 1* 2 8 active quorum node 2 siccps10 1* 2 8 active quorum node 3 siccps17 1* 2 8 active 5 siccps13 1* 2 8 active 6 siccps14 1* 2 8 active 7 siccps15 1* 2 8 active 8 siccps16 1* 2 8 active 9 siccps18 1* 2 8 active
Page 44 of 156
3.2.2 SAN Storage The SAP NetWeaver Landscape Virtualization Management allows the direct management of natively attached storage without having the need of a shared file system like GPFS. IBM provides a deep integration of IBM storage functionality with SAP LVM. This includes, but is not limited to, creation and mapping of storage volumes to hosts or LPARs discovering the volumes within the operating system create file systems creation of snapshots of volumes mapping the snapshots to hosts / LPARs re-mapping of volumes This allows automated and fast moving, cloning and copying of SAP systems. This chapter briefly describes the storage components and setup used in this proof of concept. You can find the detailed description about how the components were installed and configured in chapter 3.4.4.
3.2.2.1 Storage Components
Storage systems: IBM SAN Volume Controller IBM XIV Storage System Firmware Version: 10.2.4a 79 TB usable space A separate pool ISICC_cloud was created for the volumes of the test environment Storage Management Software: XIV CLI version (XCLI) version: 3.1 build 18 IBM Tivoli FlashCopy Manager: 3.2.0.1
3.2.2.2 SAP LVM SAN Configuration and File Systems
You need to install the SAP instances to be managed with SAP LVM according to the SAP installation guides. The following file system layout is recommended, and required if IBM Tivoli Storage FlashCopy Manager is used to create a FlashCopy backup or clone of a SAP DB instance; typically this applies to SAP production systems. The number of sapdata file systems (see chapter 3.4.5.1) might vary, the default is four. A minimum of three different volume groups need to be configured, as shown in Figure 11.
Page 45 of 156
Figure 11 Volume Group and Filesystem layout List of file systems used:
root> mount node mounted -------- --------------/dev/hd4 /dev/hd2 /dev/hd9var /dev/hd3 /dev/hd1 /dev/hd11admin /proc /dev/hd10opt /dev/livedump siccserv /siccfs /dev/lv10 /dev/lv11 /dev/lv13 /dev/lv12 /dev/lv15 /dev/lv09 /dev/lv16 /dev/lv01 /dev/lv03 /dev/lv04 /dev/lv02 /dev/lv05 /dev/lv06 /dev/lv07 /dev/lv08 /dev/lv14 mounted over vfs date options --------------- ------ ------------ --------------/ jfs2 Jul 02 09:47 rw,log=/dev/hd8 /usr jfs2 Jul 02 09:47 rw,log=/dev/hd8 /var jfs2 Jul 02 09:47 rw,log=/dev/hd8 /tmp jfs2 Jul 02 09:47 rw,log=/dev/hd8 /home jfs2 Jul 02 09:48 rw,log=/dev/hd8 /admin jfs2 Jul 02 09:48 rw,log=/dev/hd8 /proc procfs Jul 02 09:48 rw /opt jfs2 Jul 02 09:48 rw,log=/dev/hd8 /var/adm/ras/livedump jfs2 Jul 02 09:48 rw,log=/dev/hd8 /home/siccfs nfs4 Jul 02 09:48 ro,bg,hard,intr,vers=4,sec=sys /sapmnt/CL6 jfs2 Jul 02 11:02 rw,log=/dev/loglv00 /usr/sap/CL6 jfs2 Jul 02 11:02 rw,log=/dev/loglv00 /home/cl6adm jfs2 Jul 02 11:02 rw,log=/dev/loglv00 /sapinst jfs2 Jul 02 11:02 rw,log=/dev/loglv00 /usr/sap/DAA/SMDA97 jfs2 Jul 02 11:02 rw,log=/dev/loglv00 /db2 jfs2 Jul 02 11:02 rw,log=/dev/loglv01 /db2/CL6 jfs2 Jul 02 11:02 rw,log=/dev/loglv01 /db2/CL6/db2cl6 jfs2 Jul 02 11:02 rw,log=/dev/loglv02 /db2/CL6/log_dir jfs2 Jul 02 11:02 rw,log=/dev/loglv03 /db2/CL6/log_archive jfs2 Jul 02 11:02 rw,log=/dev/loglv01 /db2/CL6/db2dump jfs2 Jul 02 11:02 rw,log=/dev/loglv01 /db2/CL6/sapdata1 jfs2 Jul 02 11:02 rw,log=/dev/loglv02 /db2/CL6/sapdata2 jfs2 Jul 02 11:02 rw,log=/dev/loglv02 /db2/CL6/sapdata3 jfs2 Jul 02 11:02 rw,log=/dev/loglv02 /db2/CL6/sapdata4 jfs2 Jul 02 11:02 rw,log=/dev/loglv02 /var/db2 jfs2 Jul 02 11:02 rw,log=/dev/loglv01
Page 46 of 156
List of volume groups used:

root> lsvg -l vg00 vg01 vg02 vg03 vg00: LV NAME TYPE LPs loglv00 jfs2log 1 lv12 jfs2 256 lv10 jfs2 251 lv11 jfs2 376 lv13 jfs2 8 lv15 jfs2 62 vg01: LV NAME TYPE LPs loglv01 jfs2log 1 lv14 jfs2 1 lv02 jfs2 7 lv04 jfs2 254 lv09 jfs2 178 lv16 jfs2 1 vg02: LV NAME TYPE LPs loglv02 jfs2log 1 lv01 jfs2 1 lv05 jfs2 313 lv06 jfs2 313 lv07 jfs2 313 lv08 jfs2 313 vg03: LV NAME TYPE LPs loglv03 jfs2log 1 lv03 jfs2 497
PPs 1 256 251 376 8 62 PPs 1 1 7 254 178 1 PPs 1 1 313 313 313 313 PPs 1 497
PVs 1 1 1 1 1 1 PVs 1 1 1 1 1 1 PVs 1 1 1 1 1 1 PVs 1 1
LV STATE open/syncd open/syncd open/syncd open/syncd open/syncd open/syncd LV STATE open/syncd open/syncd open/syncd open/syncd open/syncd open/syncd LV STATE open/syncd open/syncd open/syncd open/syncd open/syncd open/syncd LV STATE open/syncd open/syncd
MOUNT POINT N/A /sapinst /sapmnt/CL6 /usr/sap/CL6 /home/cl6adm /usr/sap/DAA/SMDA97 MOUNT POINT N/A /var/db2 /db2/CL6/db2dump /db2/CL6/log_archive /db2 /db2/CL6 MOUNT POINT N/A /db2/CL6/db2cl6 /db2/CL6/sapdata1 /db2/CL6/sapdata2 /db2/CL6/sapdata3 /db2/CL6/sapdata4 MOUNT POINT N/A /db2/CL6/log_dir
List of hdisks used:

root> lsvg -p vg00 vg01 vg02 vg03 vg00: PV_NAME PV STATE hdisk2 active vg01: PV_NAME PV STATE hdisk1 active vg02: PV_NAME PV STATE hdisk3 active vg03: PV_NAME PV STATE hdisk4 active
TOTAL PPs 1019 TOTAL PPs 1019 TOTAL PPs 1279 TOTAL PPs 1019
FREE PPs 65 FREE PPs 577 FREE PPs 25 FREE PPs 521
FREE DISTRIBUTION 00..00..00..00..65 FREE DISTRIBUTION 77..00..92..204..204 FREE DISTRIBUTION 00..00..00..00..25 FREE DISTRIBUTION 113..00..00..204..204
The list of volumes defined on the XIV storage system for SID CL6 is shown in Figure 12. It also shows the snapshots created during the cloning of SAP system CL6 as source system.
Page 47 of 156
Figure 12 Volumes of SAP system CL6 The volume wwn is part of the hdisk unique_id. This attribute is used to identify the hdisk on the storage subsystem, as highlighted in yellow and green:
XIV 1300360>>vol_list vol=CL6_datavg_1 -t wwn WWN 00173800016800CA root> lsattr -l hdisk3 -E -a unique_id unique_id 2611200173800016800CA072810XIV03IBMfcp
Figure 13 shows the database instance of SAP system CL6 in LVM and the according mount points.
Page 48 of 156
Figure 13 SAP System CL6 in LVM and mount points We recommend that host names defined on the storage system to which the volumes are mapped exactly match the real host/node name. The primary mapping of operating system to the host name on the storage system is done by using port IDs (WWPNs). In case this is not possible for some reason, mapping is identified by comparing the real host name of the operating system with the host name on the storage system. IBM Tivoli FlashCopy Manager (FCM) is used to manage the snapshots on the storage systems for SAP system cloning and copying purposes and must be also installed on all SAP LVM managed systems. The installation and configuration steps of FCM are described in detail in chapter 3.4.4). 3.3 Install Management Environment
3.3.1 Install SAP NetWeaver Landscape Virtualization Management Software The current SAP NetWeaver Landscape Virtualization Manager 2.0 software is a Java application that is deployed on a SAP NetWeaver Application Server Java system of usage type Basic. Please download the appropriate installation guide for the target operating system and database platform combination and follow the instructions in that guide to install a SAP NetWeaver Java System: http://service.sap.com/instguidesNW73 http://service.sap.com/instguidesNW74 As always please pay particular attention to the SAP notes mentioned in the SAP Notes for the Installation section of the installation guide. Download and deploy the SAP NetWeaver Landscape Virtualization Management software after the installation and configuration of the SAP Java System is complete. Please download the latest version of the installation guide for your licensed version (standard edition or enterprise edition): http://service.sap.com/instlvm Again it is important to check all SAP notes mentioned on the download page and in the installation guide itself. The installation guide describes the necessary steps to install LVM after the initial setup of SAP NetWeaver AS Java. At a high level, the installation of a SAP LVM management system requires:
Page 49 of 156
1. Install SAP NetWeaver AS Java with usage type BASIC 2. Download the required SCA files for SAP LVM from the SAP Service Marketplace http://service.sap.com/swdc Software DownloadsBrowse our Download CatalogSAP NetWeaver and complementary productsSAP NW LANDSC VIRT MGT ENT 3. Deploy the downloaded SCA files using one of the supported implementation tools (SWPM or telnet) 4. Perform the initial setup using the configuration wizard as described in the postinstallation section The last step is described in detail on the SAP Help Portal at: http://help.sap.com/lvmstd20 http://help.sap.com/lvment20 Initial SetupCarrying Out the Initial Setup Using the Configuration Wizard Follow the procedure documented in the help portal to step through the configuration wizard. After the wizard finished successfully, you should be able to access SAP LVM using the following URL: http://<host>:<port>/lvm. 3.3.2 Install IBM Tivoli Storage FlashCopy Manager SAP LVM exploits IBM Tivoli Storage FlashCopy Manager to perform snapshot backups for file systems related to the SAP Systems. The FCM setup is done according to the FlashCopy backup of "Custom application environments" documentation. The snapshots are managed as backup versions using the IBM Tivoli Storage FlashCopy Manager management policies. The storage volumes for the target system are acquired and mounted on the target LPAR. The management agent (acsd) of Tivoli Storage FlashCopy Manager is installed centrally on one of the management LPARs. The "acsd" agent coordinates the snapshot backup operation and controls the backup flow and mediates between the other agents. The management agent provides access to the snapshot backup repository, which contains information about the valid snapshot backups and their relationships to the snapshot capable storage devices. On each host an acsgen agent is started during the SAP cloning/ SAP system copy process. This agent performs the storage related tasks on the source and target hosts.
Page 50 of 156
Figure 14 illustrates the environment:

Management Environment
AIX
Managed Environment
SAP Source IBM Tivoli FlashCopy Manager acsgen /home/sapadm/acs Storage Manager native Interface
SAP LVM
AIX
e.g. IBM XIV XCLI /opt/IBM/xiv/XIVGUI3.1/xcli
IBM Tivoli FlashCopy Manager acsd /home/sapadm/acs Storage Manager native Interface
e.g. IBM XIV XCLI 3.1 /opt/IBM/xiv/XIVGUI3.1/xcli
SAP Host Agent

/usr/sap/hostctrl
VIOS vFC (NPIV)
SAN Storage
IBM XIV, IBM SVC, or IBM Storwize V7000 Storage System Storage connected via vFC (NPIV) Compliant Storage Layout
SAP Target
AIX
SAP Host Agent

/usr/sap/hostctrl
IBM Tivoli FlashCopy Manager acsgen /home/sapadm/acs Storage Manager native Interface
e.g. IBM XIV XCLI /opt/IBM/xiv/XIVGUI3.1/xcli
IBM XIV SVC / Storwize

VIOS
Source Target
SAP Host Agent

/usr/sap/hostctrl
vFC (NPIV)
Figure 14 Overview of the FlashCopy Manager Installation The Tivoli FlashCopy Manager installation will follow the same steps as described in chapter 3.4.4.8 for the setup for a "Custom application environments". Additionally, a profile template needs to be created on the management node. The following shows a sample profile for the IBM XIV Storage:
>>> GLOBAL # ACS_DIR /home/sapadm/acs ACSD siccplvm 57328 # TRACE NO <<< >>> ACSD ACS_REPOSITORY /home/sapadm/acs/acs_rep # REPOSITORY_LABEL TSM <<< >>> DEVICE_CLASS STANDARD COPYSERVICES_HARDWARE_TYPE XIV # CLONE_DATABASE NO #STORAGE_SYSTEM_ID 3984 PATH_TO_XCLI /opt/IBM/xiv/XIVGUI3.1 COPYSERVICES_SERVERNAME siccxiv COPYSERVICES_USERNAME saplvm BACKUP_HOST_NAME siccps98 # RECON_INTERVAL 12 # GRACE_PERIOD 24 # USE_WRITABLE_SNAPSHOTS AUTO # USE_CONSISTENCY_GROUPS YES <<< >>> CLIENT LVM_FREEZE_THAW NO ENHANCED_PARTITIONING NO <<<
Page 51 of 156
SAP LVM takes this template and generates runtime templates during the actual execution. The ACSD entry in the GLOBAL section of the template refers to the hostname of the management system running the FCM acsd agent. SAP LVM will replace the host name and port number (2nd argument) dynamically during the actual run. IBM Tivoli Storage FlashCopy Manager traces can be enabled in this section (set parameter TRACE YES). The ACS_REPOSITORY in the ACSD section refer to the repository directory and will be dynamically replaced too. In the DEVICE_CLASS section, ensure that the following settings are included for IBM XIV storage: COPYSERVICES_HARDWARE_TYPE needs to be set to XIV PATH_TO_XCLI needs to point to the installation directory of the XIV XCLI (see also chapter 3.4.4.7) COPYSERVICES_SERVERNAME needs to include the hostname / IP address of the XIV system COPYSERVICES_USERNAME needs to contain an administrative user on the IBM XIV system. IBM Tivoli Storage FlashCopy Manager uses this user to communicate with the IBM XIV System. The user needs to be created on the IBM XIV system first, and the password needs to be entered later during FCM setup. In the CLIENT section, two entries need to be specified:
LVM_FREEZE_THAW NO ENHANCED_PARTITIONING NO
Tivoli Storage FlashCopy Manager 3.2 uses IBM Global Security Kit (GSKit) for secure communication between the distributed processes running on different LPARs. GSKit is part of the Tivoli Storage FlashCopy Manager 3.2 software package and needs to be installed on all LPARs. After the configuration of Tivoli Storage FlashCopy Manager there are additional files (server key database, client certificate) which need to be distributed across all the LPARs involved in the scenario (central system, source system, target system). 3.3.3 LDAP Server
3.3.3.1 User Management
A SAP LVM infrastructure requires a method to synchronize naming services on all enabled servers. This includes information on users, groups, protocols, services, hosts, etc. Possible solutions are NIS/NIS+ or the synchronization of local files (typically stored in /etc) using tools like rdist. However, development of NIS has stopped and the recommended solution for centralized user management is LDAP (Lightweight Directory Access Protocol). LDAP is widely used and has become the standard protocol for access to information registries, optimized for read mostly access. AIX clients can work with any LDAP server as long as it is V3 conform. In our sample implementation, we implemented the IBM Tivoli Directory Server V6.3 as LDAP server.
More information is available at: http://www-01.ibm.com/software/tivoli/products/directory-server/ The following introduces some of the LDAP terminology and discusses briefly some important steps for planning and running LDAP on AIX. This is no replacement for the product documentation. Especially an enterprise wide usage of LDAP sharing directory information for other services requires careful planning and you need to consider issues like namespaces, secure authentication using SSL, and implementing high availability using server replication. These topics are fully covered in the product documentation and the available Redbooks and whitepapers.
3.3.3.2 Schema
A schema defines which objects and attributes can be stored in a Directory Information Tree (DIT). It defines the names of objects and attributes, syntax, and possible values as well as whether attributes are required or optional for an object. For example, an object could be people or server and attributes of the objects could be name, homedirectory, uid or ipaddress, hostname. The following schemas are available for AIX user management and authentication: AIX The AIX schema includes the aixAccount and aixAccessGroup object classes. This schema offers the entire AIX user and group attributes. This schema is included to support legacy LDAP installations prior to Version 5.2. RFC2307 The RFC2307 schema includes the posixAccount, posixGroup and other NIS related object classes. This experimental RFC defines a schema that allows NIS maps to be imported into LDAP. RFC2307 only defines a subset of the AIX user and group attributes. This schema supports any RFC2307 compliant platforms and AIX 5L Version 5.2. RFC2307AIX The RFC2307AIX schema includes the RFC2307 schema plus the AIX specific object classes, aixAuxAccount and aixAuxGroup. The AIX specific object classes provide attributes to store additional attributes not defined by the RFC2307 standard. The recommended schema for AIX5.2 and higher is RFC2307AIX and has been used for the SAP LVM reference implementation. This has several advantages such as RFC2307 is considered a standard for UNIX systems including Linux.
3.3.3.3 Namespace
The LDAP naming model defines how entries are identified and organized. Entries are organized in a tree-like structure called the Directory Information Tree (DIT) based on their distinguished name (DN). A DN is a unique name that unambiguously identifies a single entry. Distinguished Names are made up of a sequence of relative distinguished names (RDN). Each RDN in a DN corresponds to a branch in the DIT leading from the root of the DIT to the directory entry. A RDN typically has the form <attribute name> = <value>. A DN is composed of a sequence of relative distinguished names separated by commas. The following DIT shows a sample DIT with a suffix o=isicc,c=de:
c=de
o=isicc
cn=user
cn=group
uid=c01adm
gid=sapsys
uid=orac01
gid=dba
Figure 15 Sample LDAP Directory Information Tree for AIX user management
3.3.3.4 LDAP Server Installation and Configuration
For LDAP server implementations please review and follow the instructions in the mentioned references.
3.3.3.5 LDAP Client Installation
AIX LDAP client is available in a 32-bit and a 64-bit version. Their filesets are shipped with the AIX CDs and should be included in the install images for the SAPL LVM client nodes.
3.3.3.6 LDAP Client Configuration
mksecldap is an AIX command for IBM Directory server and client setup. Please check the man pages or the online documentation for the full syntax description. The mksecldap command performs the required updates in various configuration files, starts the secldapclntd daemon, and puts an entry in /etc/inittab so that the seclpdapclntd daemon will be automatically started after reboot. The following command is used in our demo environment to configure an AIX LDAP client:
/usr/sbin/mksecldap -c -h siccldap1,siccldap2 -a cn=root -p ldappw
There are a number of commands available to control the client daemon:

start-secldapclntd stop-secldapclntd restart-secldapclntd ls-secldapclntd flush-secldapclntd 3.3.3.7
start ldap client daemon stop ldap client daemon restart the ldap client daemon show the status of the ldap client daemon clear the cache of the ldap client daemon
User Authentication
The last step required to setup user login through LDAP is to change a users authentication mechanism. User authentication is controlled by the SYSTEM attribute in the /etc/security/user file. For example to enable LDAP user c01adm to login to the local system, you could execute the following command;
Page 54 of 156
IBM SAP International Competence Center chuser SYSTEM=LDAP registry=LDAP c01adm
To allow all LDAP users to login to the system, one can set each LDAP user's SYSTEM attribute to LDAP, or simply set the default stanza's SYSTEM attribute to LDAP. This has to be done manually in the /etc/security/user file, where you need to change the following two attributes in the default stanza.
Default: SYSTEM = "LDAP or compat" registry = LDAP
We recommend using local authentication for the root user so that you can still login to the server in case there is something wrong with the LDAP setup. With local authentication the root users stanza should look like:
root: SYSTEM = "compat" registry = files
In our sample environment the complete client setup is automated in a NIM script. The script uses mksecldap to configure the client and then runs a Perl command to automatically change the default and root users stanza to the appropriate: Sample Customizing Script for Automated LDAP Client Setup
#Setup LDAP Client #AIX users are managed by LDAP /usr/sbin/mksecldap -c -h siccldap1,siccldap2 -a cn=root -p ldappw perl -p -i -e 'if ( /default:/ .. /SYSTEM =/ ) { s/"compat"/"LDAP or compat"\n\tregistry = LDAP/; } elsif ( /root:/ .. /SYSTEM =/ ) { s/"compat"/"compat"\n\tregistry = files/; }' /etc/security/user 3.3.3.8 Create Users and Groups
The AIX mkuser command is able to create a user on the LDAP server using the load module option -R LDAP. Unfortunately the smit mkuser panel doesnt allow for this option. You should create a new user using the mkuser command. Once a user is created, you can use the smit chuser panel to change user attributes. For example to create a user c01adm, use the command
mkuser R LDAP registry=LDAP c01adm
Then use the smit chuser panel to change the other required attributes (of course, you could also set them directly using the appropriate attribute options with the mkuser command). The problem also applies to the mkgroup command, so you have to create the required groups manually as well:
mkgroup R LDAP sapsys
Consider using some naming conventions for the numerical user id, for example by reserving certain ranges for the SAP and database administration users:

mkgroup mkgroup mkgroup mkgroup mkgroup mkgroup -R -R -R -R -R -R LDAP LDAP LDAP LDAP LDAP LDAP registry=LDAP registry=LDAP registry=LDAP registry=LDAP registry=LDAP registry=LDAP id=400 id=401 id=402 id=403 id=404 id=405 dbqa1adm dbqa1ctl dbqa1mnt dbqa1mon sapsys sapinst
mkuser -R LDAP registry=LDAP id=400 pgrp=sapsys groups='sapsys,dbqa1ctl,dbqa1mon,sapinst' home='/home/qa1adm' gecos='SAP System Administrator' shell='/usr/bin/csh' fsize=-1 cpu=-1 data=-1 stack=-1 core=2097151 qa1adm mkuser -R LDAP registry=LDAP id=401 pgrp=dbqa1adm groups='dbqa1adm' home='/db2/db2qa1' gecos='SAP Database Administrator' shell='/usr/bin/csh' fsize=-1 cpu=-1 data=-1 stack=-1 core=2097151 db2qa1 mkuser -R LDAP registry=LDAP id=402 pgrp=dbqa1mnt groups='dbqa1mnt' home='/home/sapqa1' gecos='ABAP Database Connect User' shell='/usr/bin/csh' fsize=-1 cpu=-1 data=-1 stack=-1 core=2097151 sapqa1 mkuser -R LDAP registry=LDAP id=403 pgrp=dbqa1mon groups='dbqa1mon' home='/home/sapqa1db' gecos='Java Database Connect User' shell='/usr/bin/csh' fsize=-1 cpu=-1 data=-1 stack=-1 core=2097151 sapqa1db
3.3.4 Other Central Management Components

3.3.4.1 OS Management
We highly recommend a centralized OS software management concept. There are two common solutions to manage the installation and maintenance of AIX images in a server landscape. The first one is using the AIX built-in Network Installation Management (NIM) features. The second one is to use the VMControl plugin of IBM Flex System Manager or IBM Systems Director. NIM allows the installation and maintenance of AIX base operating system and additional software from a central repository. As images are sent across the network to the partition to be installed, the installation steps can be performed fully remotely without needing physical access to the server. During and after base installation, you can customize the machines by running specific customizing scripts. Later on, you can deploy fixes and new software images to the server partitions from a central administration point. NIM supports several different machine types: diskless, dataless and standalone. A diskless machine has no own disks, all file systems and other resources are installed remotely and accessed via the network. A dataless machine has local paging and dump devices, the file systems for / and /usr are located on a remote server, all the other file systems are located either local or remote. On a standalone machine, all file systems and resources are installed on local disks. Due to performance and availability reasons, we recommend using local (or SAN attached disks) on the computing nodes and install them as standalone machines. It is possible to use diskless or dataless machines for test and development systems. They boot of the network and mount the OS from the NIM server. The following two Redbooks describe in detail the concepts of centralized OS management using NIM or VMcontrol: NIM from A to Z in AIX 5L http://www.redbooks.ibm.com/abstracts/sg247296.html?Open IBM Systems Director VMControl Implementation Guide on IBM Power Systems http://www.redbooks.ibm.com/abstracts/sg247829.html?Open
Page 56 of 156
IBM SAP International Competence Center 3.3.4.2 Network Infrastructure
There are no special network infrastructure requirements for an SAP LVM landscape. Common standard today and recommended environment is a switched Gigabit Ethernet network infrastructure. The major difference compared to a traditional SAP implementation is the consistent usage of virtual IP names and addresses assigned to a SAP instance. These addresses need to be implemented in the customers name services infrastructure, typically based on DNS (please refer to the section 3.1. for some recommendations about naming conventions for virtual IP names). If virtual network addresses come from new subnets, one might have to update router tables, saprouter definitions, etc. to ensure that the end user can access the SAP instance using the virtual IP name. You also have to change SAP logon groups to reflect the virtual IP names. Note 962955 - Use of virtual TCP/IP host names 3.4 Install System Landscape The following section provides an overview about the preparation steps required for the environment. This includes considerations about PowerVM and VIOS IBM Flex System Manager or IBM Systems Director IBM HMC OS setup for the hosts SAP Installation
3.4.1 PowerVM and VIOS Base assumption is that the volume(s) for the AIX operating system (rootvg) is/are located on the same storage system as the SAP system, and that the rootvg storage volumes are also attached via virtual fibre channel (vFC). In this case, the host and adapter definition in the storage system required for the access to the rootvg volumes will ensure the required connectivity to the SAP application related volumes in the storage system also. 3.4.2 IBM Systems Director Setup The following describes high-level configuration steps required for the integration of either IBM Systems Director or IBM Flex System Manager with SAP LVM. Further details are described in SAP Note 1728222 - Configuration of IBM Systems Director VMControl for SAP LVM The term virtualization manager is applied for both integration scenarios. IBM Systems Director and IBM Flex System Manager both require the identical steps for the setup. The tasks on the virtualization manager include: Discover the IBM HMCs on the virtualization manager and configure access to them. Perform a "collect inventory" task on the virtualization manager for IBM HMC, Power servers, and for all logical partitions (LPARs).
Page 57 of 156
Discover operating systems running on partitions, configure access, and perform "collect inventory" task on virtualization manager for all the discovered operating systems. Define a user for SAP LVM access on the virtualization manager The default administrator user can be used as user for SAP LVM. The administrators in SAP LVM then have the same credentials and get data for all the IBM Power Systems configured in the virtualization manager. To restrict the access to the specific set of IBM Power Systems managed by SAP LVM, a specific user can be created in the virtualization manager. This user does not have the credentials for all Power systems available in the virtualization manager, but only to those in scope of the SAP LVM environment. The configuration is described in chapter 4.9.
Create a security certificate on the virtualization manager Per default, the virtualization manager uses Port 8422 for communication. Communication via this port must be allowed in the network and firewall settings for the environment. o Open the URL https://<virtualization manager host>:8422" in a browser window. During the first connection attempt the browser displays a notification window to accept the certificate. This virtualization manager certificate is stored in the internet browser configuration locally, and can be exported to a certificate file. o The virtualization manager certificate needs to be imported into the SAP NetWeaver system with the deployed SAP LVM software. The certificate store must be "TrustedCAs". Check that the hostname in the certificate is valid.
For more information on this topic see also the SAP documentation "Using the AS Java Key Storage". Using the virtualization manager for AIX OS provisioning requires to discover, to access and to perform "collect inventory" tasks for NIM, VIO servers, fibre channel switches and storage managers in addition. It must be possible to Capture and deploy an AIX operating system via NIM (4.9) Select a particular IBM Power server as target for deployment Select a storage system for deployment within the different layers of the virtualization manager (VMControl, Storage Manager or Control). Then similar provisioning operations can be triggered out of SAP LVM. Using the virtualization manager for LPAR relocation (Live Partition Mobility) requires the configuration of server pools or virtual farms. If it is intended to migrate LPAR "X" hosted on IBM Power system "A" to IBM Power system "B", then
Page 58 of 156
both Power system "A" and Power system "B" must be members of the same server pool or virtual farm. If the virtualization manager is able to relocate a partition, the same operation can be triggered in SAP LVM too. 3.4.3 HMC Setup The recommended setup for SAP LVM is to use IBM Flex System Manager or IBM Systems Director as virtualization manager. However, the management via IBM Power HMC is possible too. This is described in detail in SAP Note 1728293 - Configuration of IBM Power HMC for SAP LVM. 3.4.4 Install Hosts This section describes the base configuration steps on an AIX operating system level for the compute nodes. All the required steps need to be executed at least on the first system image. This image can then be captured (NIM, VMControl) and can be used as a "master image template for the deployment of all additional compute nodes. The required steps include Activate IPsec Check/ update DNS configuration Configure the LDAP client Adapt /etc/services base configuration Check/adapt filesystem sizes Install SAP Host Agent package and SAPACEXT libraries Install XIVGUI (IBM XIV Storage systems) Configure ssh (IBM SVC or Storwize systems) Install IBM Tivoli Storage FlashCopy Manager
3.4.4.1 Activate IPsec
Each of the hosts participating in a system provisioning use case for SAP system cloning or SAP system copy needs to be prepared for network isolation. In an AIX environment, the network isolation is provided via IPsec. IP security, commonly known as IPsec, is a protocol developed by the Internet Engineering Task Force (IETF), designed to provide "end-to-end" cryptographically based security for IP network connections. IPsec implementations include a method of restricting connections to various services, based on their origin and destination. This feature is known as packet filtering. All packets on an IP network originate from an IP address and a port, and are destined for another IP address and port. A packet filter is a physical or virtual device that sits between the endpoints of a connection and determines whether the packet should be permitted to continue to its destination. The decision is made by comparing various attributes of the packet to rules that are defined by the administrator of the packet filter. Those attributes include source address and subnet, source port, destination address and subnet, destination port, protocol, direction of the connection, and fragmentation of the packet. Make sure that APAR IZ98741 "DYNAMIC FILTER RULE ID NOT MODIFIED BY RMFILT -N ALL" is installed in the AIX OS image IPsec is activated via the AIX SMIT tool using fast path ips4_start:
Page 59 of 156
IBM SAP International Competence Center # smitty ips4_start
Figure 16 Start IPsec Alternatively, IPsec can be started directly via the command line:
# /usr/sbin/mkdev -c ipsec -t 4
No manual definitions are required for packet filters. SAP LVM defines or releases automatically the packet filters for system isolation during the execution of the System Copy or System Clone workflows. Figure 17 illustrates an additionally required step during the definition of a managed host. The checkbox Isolation Ready must be enabled so that the LPAR can participate as a target host in a System Copy/System Clone scenario:
Figure 17 Define the Host as Isolation Ready in SAP LVM

3.4.4.2 DNS Name Resolution
Ensure that the search order for DNS name resolution is first based on the local /etc/hosts file, and then on the external DNS nameserver.
Page 60 of 156
IBM SAP International Competence Center # vi /etc/netsvc.conf hosts = local,bind4
In our test environment, the content of the /etc/hosts file contains only the IP label for the hostname of the partition. All external addresses are resolved via DNS. In this case, communication to the DNS server needs to be allowed during the isolated, network fenced state within the System Clone / System Copy workflow. The required settings to allow DNS communication are illustrated in chapter 4.7, Figure 42.
3.4.4.3 Configure LDAP Client
As prerequisite for the use of LDAP the IBM Tivoli Directory Server Client needs to be installed on each LPAR. idsldap.clt32bit61 idsldap.clt64bit61 The LDAP client needs to be defined on the LPAR in the same way as described in chapter 3.3.3.5 and 3.3.3.6.
3.4.4.4 Adapt /etc/services Entries
Table 8 illustrates the ports to be predefined for ABAP instances. For simplicity all ports covering the full instance number range from 00 to 99 are added to /etc/services. In total 400 ports for sapdpXX, sapgwXX, sapdpXXs and sapgwXXs are added. If only a subset of the ports is required in the organization (e.g. due to SAP instance number standards) then it is sufficient to add only those specific ports to the /etc/services file.
Port sapdpXX XX { 00, 01, 02, 99 } sapgwXX XX { 00, 01, 02, 99 } sapdpXXs XX { 00, 01, 02, 99 } sapgwXXs XX { 00, 01, 02, 99 } Description SAP System Dispatcher Ports /etc/services content sapdp00 3200/tcp sapdp99 3299/tcp sapgw00 3300/tcp sapgw99 3399/tcp sapdp00s 4700/tcp sapdp99s 4799/tcp sapgw00s 4800/tcp sapgw99s 4899/tcp
SAP System Gateway Ports
SAP System Dispatcher Security Ports
SAP System Gateway Security Port
Table 8 Entries in /etc/services

3.4.4.5 Adapt File System Sizes
Several software components need to be available on each of the LPARs in the managed environment. If all the software components are installed locally on the LPAR, then sufficient free space is required in certain filesystems for the management components. Table 9 gives an example about required sizes for the components in the local filesystems/directories:
Directory /usr/sap/hostctrl /opt/IBM/xiv/XIVGUI3.1 (example) Size
(approx.)
VG rootvg or saplocalvg rootvg
Content SAP Host Agent package, together with the SAPACEXT libraries XIV XCLI Installation
MByte 200 350
Page 61 of 156
IBM SAP International Competence Center Directory /usr/tivoli/tsfcm /home/sapadm/acs (example) Size
(approx.)
VG rootvg rootvg or saplocalvg
Content Installation directory for IBM Tivoli Storage Work directory for IBM Tivoli Storage FlashCopy Manager (SAP LVM context)
MByte 700 700
Table 9 Local Filesystem layout Increase the size of the /usr filesystem to allow to install the SAP Host Agent package and the IBM Tivoli Storage FlashCopy Manager software (FCM) Increase the size of the /home filesystem for the copy of the FCM package in /home/sapadm/acs Increase the size of the /opt filesystem for the XIVGUI package
# chfs -a size=+1500M /usr # chfs a size=+1G /home # chfs -a size=+1G /opt
Page 62 of 156
IBM SAP International Competence Center 3.4.4.6 Install SAP Host Agent and SAP Adaptive Computing Extensions
The runtime user for the SAP Host Agent is user sapadm. SAP LVM uses this user id to authenticate with the SAP Host Agent on the managed system. The user sapadm must be available on a compute node before the SAP Host Agent installation. The SAP Host Agent package requires root user credentials during installation. The group sapsys must exist on the host. The user sapadm must exist and be a member of the sapsys group. Create a home directory for sapadm. The FCM setup on the managed LPAR is performed in the home directory of user sapadm
# mkdir /home/sapadm # chown sapadm.sapsys /home/sapadm
Download the recent SAPHOSTAGENT.SAR archive and unpack it using the SAPCAR tool. The archive contains the saphostexec program. The SAP Host Agent package is installed/upgraded by executing saphostexec install, respectively saphostexec upgrade as root user. This installs the SAP Host Agent package in directory /usr/sap/hostctrl. Some SAP Host Agent runtime executables have the setuid bit set allowing for the execution of privileged commands during SAP LVM prepare and unprepare operations. Download the SAPACEXT.SAR archive and unpack it to the operations.d subdirectory of the SAP Host Agent installation.
# cd /usr/sap/hostctrl/exe/operations.d # SAPCAR xvf /tmp/SAPACEXT.SAR
Additional considerations for the SAP Host Agent are found in the SAP Note Note 1292836 - Logfiles of the SAP Host Agent
3.4.4.7 Install XIVGUI
In the proof of concept environment XIVGUI 3.1 build 18 is installed. The XIVGUI image is deployed to the directory /opt/IBM/xiv (XIVGUI is part of the OS image).
Parameter XCLI Installation Directory Value /opt/IBM/xiv/XIVGUI3.1
Table 10 XIVGUI Install directory At first create the target directories for the XIVGUI image
# mkdir p /opt/IBM/xiv/XIVGUI3.1
Page 63 of 156
Install additional rpm packages for the XIVGUI image. Although this is a prerequisite this is not explicitly mentioned in the XIV XCLI installation guide: Required rpm packages include: o libgcc o libstdc o readline-5 readline-5 additionally requires o bash o info
# # # # # # rpm rpm rpm rpm rpm rpm -i -i -i -i -i -i libgcc-4.2.0-3.aix6.1.ppc.rpm libstdc++-4.2.0-3.aix6.1.ppc.rpm gcc-4.2.0-3.aix6.1.ppc.rpm bash-4.2-6.aix5.1.ppc.rpm info-4.6-1.aix5.1.ppc.rpm readline-5.2-3.aix5.1.ppc.rpm
Adjust the library settings and the LIBPATH to include these shared libraries
# cd /opt/freeware/lib # ln -s /opt/freeware/lib/gcc/powerpc-ibm-aix6.1.0.0/4.2.0/libgcc_s.a # ln -s /opt/freeware/lib/gcc/powerpc-ibmaix6.1.0.0/4.2.0/libstdc++.a # export LIBPATH=$LIBPATH:/opt/freeware/lib
Extract the XIVGUI software from the Software Image Repository

# ./xivgui-3.1-build18-aix.bin
Adapt the xcli script There is a bug in the xcli script: xcli can be started only out of the installation path. xcli needs to be patched to be able to start it from any location. Edit the xcli script and add the line
prg="$0"
in the upper section of the xcli script Validate the XCLI by a test connection to the IBM XIV storage system
# /opt/IBM/xiv/XIVGUI3.1/xcli User Name: ... Password: ... Machine IP/Hostname: ... connecting. XIV 1300360>>
Page 64 of 156
IBM SAP International Competence Center 3.4.4.8 Install IBM Tivoli Storage FlashCopy Manager
SAP LVM exploits IBM Tivoli FlashCopy Manager for the backup of "Custom application environments". On the managed systems, the acsgen agent is used to invoke the different storage related activities on the LPAR during the System Copy/ System Cloning process. Base Installation of IBM Tivoli Storage FlashCopy Manager ./3.2.0.1-TIV-TSFCMFTP-AIX.bin
Parameters: - Choose Locale: 2- English
- PRESS <ENTER> TO CONTINUE - Press Enter to continue viewing the license agreement, or enter "1" to accept the agreement - Choose the Install Set to be installed by this installer: 4 - IBM Tivoli Storage FlashCopy (R) Manager (Custom Applications) - ENTER AN ABSOLUTE PATH: /usr/tivoli/tsfcm/acs_3.2.0.1 - IS THIS CORRECT? (Y/N): Y - PRESS <ENTER> TO CONTINUE - PRESS <ENTER> TO EXIT THE INSTALLER
Check the license file (contained in IBM Tivoli FlashCopy Manager base installation). If the FCM 3.2.0.1 package was downloaded as an update package, it will not include the license file. The license file included in the FCM 3.2.0.0 base package needs to be copied to the directory: o Copy license file tsmacs.lic from the FCM 3.2.0.0 base package to the FCM install directory /usr/tivoli/tsfcm/acs_3.2.0.1 Create Directory for FCM profile / environment:
# # # # mkdir chown mkdir chown /home/sapadm sapadm:sapsys /home/sapadm /home/sapadm/acs sapadm:sapsys /home/sapadm/acs
Run the setup for FCM custom application

# cd /usr/tivoli/tsfcm/acs_3.2.0.1 # /setup_gen.sh Script not started from 'acs' sub-directory of the Home directory of a custom application user. Trying to locate the correct Home directory or install IBM Tivoli Storage FlashCopy(R) Manager in a new Home directory ... Please enter the Home directory: /home/sapadm
Page 65 of 156
IBM SAP International Competence Center This is a new installation of IBM Tivoli Storage FlashCopy(R) Manager into this Home directory. Please enter the name of the user of this Home directory: sapadm Please enter the group name of the user: sapsys
Installing/Updating IBM Tivoli Storage FlashCopy(R) Manager installation for /home/sapadm/acs Checking environment - - - - - - - - - - + OK: installation directory exists. + OK: setup_gen.sh runs in IBM Tivoli Storage FlashCopy(R) Manager installation directory '/usr/tivoli/tsfcm/acs_3.2.0.1' + OK: IBM Tivoli Storage FlashCopy(R) Manager installation will be installed to '/home/sapadm' for the first time. Installation messages - - - - - - - - - - + OK: New installation ! + OK: IBM Tivoli Storage FlashCopy(R) Manager directory '/home/sapadm/acs' created successfully. + OK: Files successfully copied.
checking /home/sapadm/acs/acsgen ... OK Select one of these configurations: (1) On-Site Production System configuration with optional remote Backup System configuration (2) On-Site Backup System configuration Enter '1' to configure IBM Tivoli Storage FlashCopy(R) Manager on the production server with the option to remotely synchronize the configuration of one or multiple backup systems using Secure Shell. Enter '2' to configure IBM Tivoli Storage FlashCopy(R) Manager on the backup system as a 'separate installation'. 2 Going to install on backup server SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure Copyright IBM Corporation, 2013 Page 66 of 156
IBM SAP International Competence Center Please enter the directory where the profile will be stored: [/home/sapadm/acs]
Deactivating IBM Tivoli Storage FlashCopy(R) Manager management daemon.
Are you going to perform off-loaded backups to Tivoli Storage Manager? [y|n] n Profile configuration ===================== Special commands: '?' ==> Show help for current parameter '!d' ==> Delete current parameter instance (only applicable to multi-instance parameters)
FMM8422I Creating new profile '/home/sapadm/acs/profile' for application 'GENERIC' ... ****** Profile parameters for section GLOBAL : ****** Path of the IBM Tivoli Storage FlashCopy(R) Manager directory {ACS_DIR} = [/home/sapadm/acs] Hostname and port of machine running Management Agent {ACSD} (<hostname> <port>) = [] siccplvm 57328 Low level tracing {TRACE} (YES|NO) = [NO]
FMM8426I Saving profile '/home/sapadm/acs/profile' ... FMM1555I Profile successfully created.
Please enter the password for authentication with the ACS daemon: Please re-enter password for verification:
Creating password file at /home/sapadm/acs/shared/pwd.acsd. A copy of this file needs to be available to all components that connect to acsd. Enter the device classes to use for this Backup System. Make sure each device class is separated by a comma. SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure Copyright IBM Corporation, 2013
Page 67 of 156
IBM SAP International Competence Center STANDARD IBM Tivoli Storage FlashCopy(R) Manager requires at least two daemon processes to be running at all times. IBM Tivoli Storage FlashCopy(R) Manager can add the necessary daemon processes to the inittab. Alternatively you may choose to start and stop these processes yourself, for instance if you want to include the processes in your HA policy. Do you want IBM Tivoli Storage FlashCopy(R) Manager to create the inittab entries for you? [y|n] n Please run the following daemon processes: /home/sapadm/acs/acsgen -D -M -s STANDARD Please make sure that the command '/home/sapadm/acs/tsm4acs' is started by a scheduler or manually.
No special customization for the FCM profile is required on the host. SAP LVM will derive the required content for the profile out of the master profile on the FCM management LPAR (acsd), and copy the profile to the source and target hosts during the execution of the SAP System Cloning/SAP System Copy workflow. No special customization for the FCM profile is required on the host. Install Global Security Kit IBM Global Security Kit (GSKit) needs to be installed on all LPARs participating in the scenario. GSKit can be installed using the setup script:
# cd /home/sapadm/acs # ./setup_gen.sh -a install_gskit -d /home/sapadm
Afterwards, check if the filesets are properly installed: lslpp -l GSKit8.gskcrypt64.ppc.rte lslpp -l GSKit8.gskssl64.ppc.rte Distribute certificate files Distribute the files for the Server Key DB o fcmcert.crl o fcmcert.rdb o fcmcert.kdb o fcmcert.sth and the Client Certificate o fcmselfcert.arm across all LPARs (directory /home/sapadm/acs in our case). Tivoli Storage FlashCopy Manager password file The FlashCopy Manager configuration on the central node (acsd) generates a password file in /home/sapadm/acs/shared/pwd.acsd. The password file needs to be copied to all the FlashCopy Manager directories on the target hosts. It contains the
Page 68 of 156
connection password to the acsd, and also the connection password to the Storage System. A NFS-shared directory for the ACS directory would simplify the setup for the password file and the certificates, but was not realized in the lab environment. 3.4.5 SAP System In the following section, base considerations for the installation of a new SAP system, as well as the eventually required modifications of an existing SAP system are given. These considerations need to be taken into account, before the SAP system then can be registered with SAP LVM. The definition of the new system in the SAP LVM context is described in chapter 4.7
3.4.5.1 Filesystem considerations for the SAP Systems (SAP System Copy) DB2 UDB LUW
3.4.5.1.1
The two tables below illustrate two different storage layouts for a SAP System based on DB2 UDB. The first example (Table 11) includes a simple layout, having a few different filesystems only and just separating between SAP application and database filesystems. The filesystem layout fulfils the requirements for the relocation and SAP system copy scenarios with SAP LVM. Place all DB2 UDB Files into one common AIX Volume Group for the database:
Filesystem /db2 Volume Group vg01 Content Contains the instance directory /db2/db2<sid> and all other database related directories. All DB2 database specific data is stored in a directory structure below the filesystem /db2 Filesystem for DB2 global registry Filesystem for home directory of SAP admin user Filesystem for SAP profile, SAP work/interface data, and SAP Kernel source Filesystem for SAP Instance directories Filesystem for SMD agent
/var/db2 /home/<sid>adm /sapmnt/<SID>
vg01 vg00 vg00
/usr/sap/<SID> /usr/sap/DAA/SMDA97
vg00 vg00
Table 11 Minimum filesystem layout (DB2 UDB) In the second example (Table 12) the filesystem layout is more complex. The database layout is split-up into different volume groups. The filesystem layout fulfils the requirements for the relocation and SAP system copy scenarios with SAP LVM as well as the capability for performing SnapShot/FlashCopy backups for the database with IBM Tivoli Storage FlashCopy manager. Separate DB2 datafiles, active logfiles, and all other DB2 files into three different AIX Volume Groups Make sure that the database directory /db2/<SID>/db2<sid> is part of the volume group with the data files
Page 69 of 156
IBM SAP International Competence Center Filesystem /db2/ /db2/<SID> Volume Group vg01 vg02 Content Contains the instance directory /db2/db2<sid>. Contains the database directory. In case that regular SnapShot backups are intended for the database, the content needs to be included in the SnapShot image together with the DB2 database containers/ storage paths Separate filesystem for DB2 inactive logs Separate filesystem for DB2 active logs: In case SnapShot backups are regularly taken and should be used for a fast restore, this filesystem *must* be separate from database data. Separate filesystem for the db2diag.log for convenience Contains the database containers / storage paths Filesystem for home directory of SAP admin user Filesystem for SAP profile, SAP work/interface data, and SAP Kernel source Filesystem for SAP Instance directories Filesystem for SMD agent Filesystem for DB2 global registry
/db2/<SID>/log_archive /db2/<SID>/log_dir
vg01 vg03
/db2/<SID>/db2dump /db2/<SID>/sapdataX /home/<sid>adm /sapmnt/<SID>
vg01 vg02 vg00 vg00
/usr/sap/<SID> /usr/sap/DAA/SMDA97 /var/db2
vg00 vg00 vg01
Table 12 Extended filesystem layout (DB2 UDB) If FlashCopy backups for the database are a requirement, then a layout similar to the second example has to be implemented.
3.4.5.1.2 Oracle Database
The two tables below illustrate two different storage layouts for a SAP System based on an Oracle database. The first example (Table 11) includes a simple layout, having a few different filesystems only and just separating between SAP application and database filesystems. The filesystem layout fulfils the requirements for relocation and SAP system copy scenarios with SAP LVM. Place all ORACLE Files into one common AIX Volume Group for the database:
Filesystem /oracle /home/<sid>adm /sapmnt/<SID> Volume Group vg01 vg00 vg00 Content Contains all database related directories. Filesystem for home directory of SAP admin user Filesystem for SAP profile, SAP work/interface data, and SAP Kernel source Filesystem for SAP Instance directories Filesystem for SMD agent
vg00 vg00
Table 13 Minimum filesystem layout (Oracle)

In the second example (Table 12) the filesystem layout is more complex. The database layout is split-up into different volume groups. The filesystem layout fulfils the requirements for relocation and SAP system copy scenarios as well as the capability for performing SnapShot/FlashCopy backups for the database with IBM Tivoli Storage FlashCopy manager: Separate Oracle datafiles, online logfiles and other Oracle files into three different AIX volume groups Relocate all non datafiles (e.g. Oracle control files) from the AIX volume group for the Oracle datafiles to a new location o The SAP DB installation per default stores one control file below /oracle/<SID>/sapdata1 Example for new control file locations: /oracle/<SID>/origlogA/cntrl/cntrl<SID>.dbf /oracle/<SID>/origlogB/cntrl/cntrl<SID>.dbf /oracle/<SID>/mirrlogA/cntrl/cntrl<SID>.dbf An additional, new filesystem is required for the redo-logs written during execution of a SAP System Copy or Clone (for the rollforward recovery of the database). This filesystem is either exported via NFS and mounted on the target host, or is also Flashcopied and mounted on the target. either exported via NFS and gets mounted on the target host, or is also Flash-copied and mounted on the target.
Volume Group vg01 vg02 Content Oracle binaries Contains the database directory. In case that regular SnapShot backups are intended for the database, the content needs to be included in the SnapShot image together with the Oracle data files Separate filesystem for Oracle archive logs Separate filesystem for DB2 online logs: In case SnapShot backups are regularly taken and should be used for a fast restore, this filesystem *must* be separated from database data.
Filesystem /oracle/<SID>
/oracle/<SID>/oraarch/ /oracle/<SID>/origlogA /oracle/<SID>/origlogB /oracle/<SID>/mirrlogA /oracle/<SID>/mirrlogB
vg01 vg03
/oracle/<SID>/saptrace /oracle/<SID>/sapreorg /oracle/<SID>/sapdataX /home/<sid>adm /sapmnt/<SID>
vg01 vg01 vg02 vg00 vg00
vg00 vg00
Contains the database containers/storage paths Filesystem for home directory of SAP admin user Filesystem for SAP profile, SAP work/interface data, and SAP Kernel source Filesystem for SAP Instance directories Filesystem for SMD agent
Table 14 Extended filesystem layout (Oracle)

If FlashCopy backups for the database are a requirement, then a layout similar to the second example has to be implemented.
3.4.5.1.2.1 Filesystem for Oracle Redo Logs for System Clone and System Copy
During the SAP LVM Step Clone Volumes with DB backup mode (execution of the storage based copy), the Oracle database is set into begin backup mode. For the duration of this mode data of changed database blocks (with full before and after information) is written to the Oracle database logs. This redo log information is required on the target system to start the database clone. When the storage based copy is complete, SAP LVM copies an Oracle DB configuration file a backup of the Oracle control file all (archived) logs written during the time frame BEGIN BACKUP until END BACKUP to the filesystem for Oracle Redo Logs. The filesystem is then accessed on the target system for the Oracle rollforward recovery to the last committed transaction of the source system. The filesystem is part of the SAP system configuration in LVM:
Figure 18 Filesystem for Oracle Redo Logs If the filesystem is shared via NFS, the following settings need to be maintained: Usage Type Oracle Redo Logs for Online Clone Storage Type NETFS Mount Point (1) JFS2 filesystem of cross-mount (2) n/a (but needs to be specified, e.g. /tmp/VCM) Export Path NFS exported fs (<NFS Server>:<NFS export> ) Mount Options hard mount, background, interrupt, read and write size 32768 (The settings are Oracle requirements for a BACKUP CONTROLFILE command to a NFS-mounted filesystem) In case of NFS sharing, SAP LVM will mount the exported directory to /tmp/VCM/<TGT_SID> on the hosts. A stanza for /tmp/VCM/<TGT_SID> is required (with all mount options) in /etc/filesystems The NFS export needs to allow root access for both the source and target host (no root squash) If multiple NFS client addresses (e.g. virtual IP addresses of DB and CI) are present in the sub-network for the NFS server access, then all of them need to be granted access in the export list on the NFS server
Page 72 of 156
IBM SAP International Competence Center 3.4.5.2 Install a new SAP system
For the SAP system installation, select one host of the SAP LVM environment that has no active SAP instances. The filesystem layout needs to be compliant to chapter 3.4.5.1 For the SAP system installation the virtual IP addresses for the database instance and the central instance/central services need to be active. Add the virtual IP addresses to the network adapter with the ifconfig command. These addresses must not be defined as persistent, as the virtual IP addresses for the SAP instances later on will be managed exclusively by SAP LVM during the prepare/unprepare workflows. The ifconfig syntax to define an alias on an existing interface is: # ifconfig <interface> alias <virtual IP> netmask <netmask> For example:
# ifconfig en0 alias c01scs netmask 255.255.255.0 # ifconfig en0 alias c01dbs netmask 255.255.255.0
During the SAP system installation, use the SAPINST_USE_HOSTNAME parameter to specify the virtual hostname for the instance to be installed. Users and Groups should be defined centrally in the LDAP environment. As SAPinst or Software Provisioning Manager (SWPM) currently cannot handle LDAP users properly, create users and groups for SAP and DB administration manually before the SAP system installation. The detailed installation procedure needs to follow all the steps described in the SAP installation documentation and the respective SAP notes.
3.4.5.3 Adapt an already existing SAP system
File System Layout The filesystem layout needs to be compliant to chapter 3.4.5.1. It may be necessary to copy or move the content of the standard installation directories into a SAP LVM compliant storage layout. Virtual IP Address and Hostname Within the SAP LVM environment the database server and the central instance/central services can be operated on separate hosts. Two different virtual IP addresses are required for them. If one of the virtual IP addresses was defined as base address of a network adapter, redefine the network adapter to a new address. Define the virtual IP addresses as IP alias to the network adapter using the ifconfig command: If the hostname for the SAP instance is changed to a new virtual hostname, make sure to adapt all occurrences of the name in the filenames and contents of DEFAULT, START, and instance profiles. Create and update the following parameters in the instance profile to the virtual hostname too.
SAPLOCALHOST = <virtual hostname CI> SAPLOCALHOSTFULL = <full qualified virtual hostname CI>
Page 73 of 156
icm/host_name_full = <full qualified virtual hostname CI>
For example, in the default profile DEFAULT.PFL, the SAPDBHOST has to reflect the virtual IP label of the DB server, and the virtual IP label of the CI will be part of the mshost, vbname, enqname, btcname, and gateway parameters:
SAPDBHOST = c01dbs rdisp/mshost = c01scs rdisp/vbname = c01pas_C01_10 rdisp/enqname = c01pas_C01_10 rdisp/btcname = c01pas_C01_10
Dependent on your individual environment, you may have to adapt settings in further interfaces/SAP systems too. (Re-) create the Users in LDAP The administrative users for Database and SAP system have to be created with the same numeric UID in the LDAP directory. If the UID for the users violate uniqueness within the SAP LVM environment, the ownership of all files needs to be changed to an unique UID. Adapt user environment files During shell startup, several files are sourced to create the proper environment for SAP admin and DB admin users. During installation time, these files are created containing the hostname within their name. Rename them in the home directory of both SAP admin and DB admin user.
# # # # # # mv mv mv mv mv mv .sapenv_<virtual hostname>.sh .sapenv.sh .apoenv_<virtual hostname>.sh .apoenv.sh .dbenv_<virtual hostname>.sh .dbenv.sh .sapenv_<virtual hostname>.csh .sapenv.csh .apoenv_<virtual hostname>.csh .apoenv.csh .dbenv_<virtual hostname>.csh .dbenv.csh
You also need to check the shell start profiles (.cshrc) that it sources all required environment files correctly. Each entry should look like
# RDBMS environment if ( -e $HOME/.dbenv_`hostname`.csh ) then source $HOME/.dbenv_`hostname`.csh else if ( -e $HOME/.dbenv.csh ) then source $HOME/.dbenv.csh endif
This means the shell first searches for a file containing the hostname, will not find it and then uses the corresponding file without the hostname. Sometimes the else if clause is missing and you need to create it manually. Adapt database configuration files to virtual IP address o ORACLE: Adapt configuration files tnsnames.ora and listener.ora
(ADDRESS = (COMMUNITY = SAP.WORLD) (PROTOCOL = TCP) SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure Copyright IBM Corporation, 2013 Page 74 of 156
IBM SAP International Competence Center (HOST = c01dbs) (PORT = 1537) )
The stanzas in tnsnames.ora and listener.ora have to reflect the virtual hostname of the DB instance. The communication port should be unique within the environment. o DB2 UDB: Update db2cli.ini to the virtual DB server address
; Comment lines start with a semi-colon. [C01] Database=C01 Protocol=tcpip Hostname=c01dbs Servicename=5912 [COMMON] Diagpath=/usr/sap/C01/SYS/global/db6/db2dump
The Hostname in the db2cli.ini file needs to reflect the virtual hostname of the DB instance. The service name reflects the communication port and should be unique within the environment. Perform a test connection to the database: Start the database (and, in case of DB ORACLE, the listener process). As user <sid>adm, perform a test connection to the database using R3trans:
c01adm> This is unicode R3trans R3trans d v R3trans version 6.22 (release 720 - 03.06.11 - 17:57:00). enabled version finished (0000).
R3trans should return 0 for a successful DB connection.

3.4.5.4 Additional SAP profile parameters
See also SAP Note 1438774 - New profile parameter system/uuid and system/description: SAP LVM will verify the system/uuid profile parameter against its configuration prior to any operation (start, stop, relocate) on any instance of a (cloned) system, to ensure that the operation is triggered on the correct instances. The system/description profile parameter contains a verbal description of the system. The parameter can be added to the SAP system later on also during the LVM configuration phase. However, the SAP system then needs to be restarted to activate the parameter.
3.4.5.5 SAP license considerations
The license key for an SAP system is checked on the partition on which the message server is running. If the message server is moved during a relocation event to another host, a license key for the new environment may be required. Request and import in advance license keys for all the systems on which the message server may run. The license key depends on the hardware key: The hardware key is different for different physical servers. (All the LPARs within one physical IBM Power server have the same hardware key).
Page 75 of 156
The license key is checked during the (re-) start of the message server: If the relocation is initialized virtualization-based (Live-Partition Mobility), then the message server keeps running in the same state as before. o Even if there is no license key for the target server available the SAP system continuous its operation. The attempt to restart the SAP system on the target server while having no valid license for the hardware key of the target server would then fail. If the relocation is initialized adaptive-based (stop and unprepare on the source host/ prepare and start on the target host), then the message server is restarted on the target host, and a valid license key for the hardware key of the target server is required. SAP has introduced a new license key feature called Flexible License Mechanism to simplify the administration of SAP license keys in system landscapes where the message server can move between several physical servers. With this method the license key is no longer tied to the hardware key of the message server. The flexible license mechanism uses a separate ID generator, which creates a unique network ID for each message server. This network ID is hardware independent and therefore it is possible to move the message server to a different host and retain the unique ID. It is possible to configure multiple ID generators to eliminate any potential single point of failure. Refer to the current SAP NetWeaver help documentation for a detailed description and instructions of how to set up this new Flexible License Mechanism: SAP NetWeaver 7.0 EHP2 and SAP NetWeaver 7.3 3.4.6 Install SAP LVM Prerequisites on the Management Node
3.4.6.1 Storage System Connectivity using SMI-S
As of November 2013 SMI-S connectivity between SAP LVM and storage system is supported for SAN Volume Controller, IBM Storwize V7000, and IBM System Storage DS8000. SMI-S connectivity is currently not supported for IBM XIV Storage Systems. Because of simplicity, SMI-S is the preferred communication method between SAP LVM and storage systems. SMI-S is already preconfigured on SAN Volume Controller and IBM Storwize V7000stem and can be used with default users. On IBM System Storage DS8000 one has to configure SMI-S. Please refer to the storage system documentation for more information.
3.4.6.2 XIV XCLI
The XIV CLI is used for the communication of LPARs with the storage system. It needs to be available on the management as well as on all managed LPARs.
3.4.6.3 SAN Volume Controller Connectivity
SMI-S is the recommended communication method for SAN Volume Controller. Nevertheless, in the lab setup we still used the SVC CLI. SAP LVM then communicates with the IBM SVC storage systems directly using ssh connectivity. The communication is initiated from the central host (the host where the acsd component of IBM Tivoli Storage FlashCopy is installed) only. First a ssh key pair has to be created for the root user on the central system. To allow the execution of CLI commands from SAP LVM the created key pair must not contain a key passphrase. The private key of the key pair is
Page 76 of 156
stored in a common directory. This location is referred as certificate later on in the SAP LVM storage manager configuration (see Figure 27). Create a RSA keypair for user root and store the keys e.g. /home/sapadm/.ssh:
> ssh-keygen -t rsa Generating public/private rsa key pair Enter file to save the key: /home/sapadm/.ssh/id_rsa Enter passphrase (empty for no passphrase): Enter passphrase again: Your identification has been saved in /home/sapadm/.ssh/id_rsa Your public key has been saved in /home/sapadm/.ssh/id_rsa.pub
An administrative user needs to be created in the SVC system. This user needs to be member of the IBM System Storage SAN Volume Controller user group "Administrator". Transfer the public key of the key pair to the IBM SVC and assign it to the administrative user.2 Perform a connectivity test after the public key is attached to the SVC admin user:
> ssh -i /home/sapadm/.ssh/id_rsa -l <SVC admin user> <SVC hostname>
See the IBM documentation IBM System Storage SAN Volume Controller and IBM Storwize V7000 Version 6.3.0 Command-Line Interface User's Guide (GC27-2287-02): Chapter 1. Preparing the SSH client for details. SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure Copyright IBM Corporation, 2013 Page 77 of 156
4 Initial LVM Configuration

4.1 Engine Settings The SAP LVM engine settings are normally configured when running the initial configuration wizard. A number of the settings (for example the default user name or password for the Host Agents) can be changed again under SetupSettingsEngine.
Figure 19 LVM Engine Settings 4.2 Virtualization Manager Configuration in SAP LVM The IBM Virtualization Manager settings (IBM Systems Director/ VMControl) are defined on the InfrastructureVirtualization Managers panel. SAP Note 1728222 Configuration of IBM Systems Director VMControl for SAP LVM describes further background information for this step. The screen shot in Figure 20 shows the start panel. Push the Add button to define a new virtualization manager in SAP LVM.
Figure 20 Virtualization Manager Configuration
Page 78 of 156
Select IBM Systems Director / IBM Flex System Manager Adapter and push the Next button.
Figure 21 Virtualization Manager: Virtualization Manager Types On the next panel enter a label for the virtualization manager, the user name, the password of the administration user that was created in chapter 4.9, the URL to access the virtualization manager, and the monitoring interval (how often SAP LVM tries to retrieve monitoring data from virtualization manager) Press Test Connection to validate all the settings and then proceed to the next panel.
Figure 22 Virtualization Manager: Virtualization Manager Properties Verify the data on the summary screen and press the Save button to store the configuration.
Page 79 of 156
Figure 23 Virtualization Manager: Summary 4.3 Storage Manager Configuration
The IBM Storage Manager settings (IBM Tivoli Storage FlashCopy Manager) are defined via the InfrastructureStorage Managers panel. The start panel is illustrated in Figure 24. Navigate to this panel, and press Add to define a Storage Manager in SAP LVM.
Figure 24 Storage Manager Configuration In the next screen (Figure 25) the Storage Manager Type can be selected. Select Vendor IBM, Product IBM Storage Adapter and click Next.
Figure 25 Storage Manager: Storage Manager type
Page 80 of 156
In the next panel (Figure 26) the IBM Tivoli Storage FlashCopy Manager related settings are specified (See also chapter 3.3.2, as a lot of these parameters need to be provided from the FCM installation): The FCM CLI Host Name field specifies the node on which the IBM Tivoli Storage FlashCopy Manager Management agent is installed. The management agent (acsd) controls the backup flow and mediates between the other agents running on the hosts. The management agent also provides access to the snapshot backup repository, which contains information about the valid snapshot backups and their relationships to snapshot capable storage devices. During a system cloning/copy process SAP LVM starts the management agent on the FCM CLI Host Name node. The FCM CLI Path specifies the working directory for the IBM Tivoli Storage FlashCopy Manager on all of the nodes. This must be the path that was specified during the installation customization of the IBM Tivoli Storage FlashCopy Manager (FlashCopy backup for generic applications in setup_gen.sh). The IBM FCM template name is the filename of the profile template located in the FCM CLI Path. Select AIX for the platform type. The Storage Connection Method specifies the storage access method from SAP LVM to the storage system.
Storage System SAN Volume Controller (V7000), or Storwize V7000 IBM XIV IBM DS8000 Storage Connection Method (SAP LVM) SMI-S (recommended) or Native CLI (SVC CLI via ssh) Native CLI (XCLI) SMI-S FlashCopy Manager communication to Storage SMI-S XCLI SMI-S
Table 15 Storage Connection Methods
Figure 26 Storage Manager: Storage Manager Properties When all required entries are defined, press Test Configuration. SAP LVM will initiate a communication test to FlashCopy Manager and the storage subsystem CLI. The configuration test should return Connection successful: SAP LVM successfully gets data from storage managers through IBM Tivoli Storage FlashCopy Manager. After a successful connection test, select Next to get to a user and password configuration screen. Here you can enter the user credentials for the configured storage system. Dependent
Page 81 of 156
on the storage system and the storage connection method you either need to provide a username and password or the path to a ssh certificate:
Storage System SAN Volume Controller (V7000), or Storwize V7000 IBM XIV IBM DS8000 Storage Connection Method (SAP LVM) SMI-S Native CLI (SVC CLI) Native CLI (XCLI) SMI-S User credentials Username/ password ssh certificate (see 3.4.6.3) Username/ password Username/ password
Table 16 Storage Connection user credentials
Figure 27 Storage Manager: Storage Systems If the user or password is changed in the storage subsystem, you have to update the LVM Storage Manager configuration too. Click Next and a summary screen appears. Verify the entered settings and click Save to save the configuration. 4.4 Configure SAP LVM Pools LVM uses pools to separate the landscape components logically and physically. Hosts and SAP systems are assigned to pools. SAP LVM operations on SAP systems assigned to a specific pool can only be targeted to hosts that are assigned to the same pool. Pools are configured on the ConfigurationPools tab. One can define multiple levels of containers and pools to reflect a hierarchical level of the logical and physical separation of the landscape entities. The screenshot below shows a container AIX Pools groups all pools with AIX based systems and hosts together. On the next level we separated the pools according to some physical attributes, all systems and hosts assigned to one pool are allocated or have physical access to the same storage subsystem.
Page 82 of 156
Figure 28 SAP LVM Pools: Pool Overview 4.5 Configure Networks
SAP LVM requires some network definitions to enable the activation and deactivation of virtual host names as aliases of existing network interfaces in a host. Navigate to InfrastructureNetwork ComponentsNetwork and click the Add button to define a new network ( Figure 32).
Figure 29 SAP LVM Network Configuration
Page 83 of 156
Enter a name, the subnet mask, and the broadcast address of the physical network on the Basic Configuration panel.
Figure 30 SAP LVM Network: Basic Configuration In the next step, Advanced Configuration settings are specified. They include the network gateway address, DNS Zone, Reverse DNS Zone, and Update IP Range. The Update IP Range can restrict the range for new IP addresses that are provisioned by SAP LVM if automatic DNS update is configured in the environment. If the "Auto Assign IP Address" option is selected in the "Host Names" step during a clone or copy operation, a random free IP address is selected from within this range. If an IP address is specified manually, it must still be within this range.
Figure 31 SAP LVM Network: Advanced Configuration Click Next to proceed to the Summary panel, verify the configuration and push the Save button to store the network configuration.
Figure 32 SAP LVM Network: Summary 4.6 Discovery of Hosts (LPARs) To discover a new host in SAP LVM, the SAP Host Agent must be installed and running on the target partition. Select ConfigurationHostsDiscover to get to the Find
Page 84 of 156
Instances/Hosts panel. There are various options to perform a host detection, in this example we select Detect Using Host and Instance Agent as Source and Hosts only under Find.
Figure 33 SAP LVM hosts: Discover Using Host and Instance Agent Enter the host name of the partition you want to discover. You can use the default credentials for the SAP Host Agent user id as configured in the engine settings or choose other authentication types like specific user/password settings or a X.509 client certificate. Click the Detect button and you should get a message that new hosts are found. On the next panel select the pool you want to assign this partition to and click the next button. Click the Save button on the summary screen to add this newly discovered host.
Figure 34 SAP LVM Hosts: Add Hosts The next step is to change some properties of this new host. Select the host on the Hosts Overview panel and click Edit. The Basic configuration step should already have the correct settings, click Next to get to the Host properties panel.
Page 85 of 156
Figure 35 SAP LVM Hosts: Edit host properties Select the AC-Enabled flag and add a network interface that is used to bind the virtual IP address of a SAP instance. You also have to provide a capacity figure (SAPS Provided) for this host. If the partition is used as a target host for cloning and system copy operations, the Isolation Ready flag has to be selected too. Press the Save button to store the changes. 4.7 Instance Configuration/ Discovery in SAP LVM The discovery for a new system is done in several steps. In the base discovery the new SAP system is discovered and added to the SAP LVM configuration. Then the configuration of the newly discovered instances is completed. This includes the detailed storage configuration for database and for the central instance.
4.7.1.1 Discovery of a New SAP System
The base discovery of new SAP systems is started on the Systems tab in the Configuration view by using the Discover button (Figure 36):
Figure 36 SAP LVM Systems: Discover new SAP instances The new system needs to meet the following prerequisites: the host is already discovered in SAP LVM the SAP Host Agent is installed and active on the host the SAP system is installed in an adaptive configuration
o virtual hostnames are defined for both database and central instance o the storage layout supports separation of database and central instance the SAP system is active all the user credentials (e.g. <sid>adm) and their passwords are known to the SAP LVM admin
For older SAP releases (without having enhanced security measures implemented) the SAP Host Agent is able to discover both database and SAP instances running on the host. For recent SAP releases the SAP Host Agent may not be able to detect the SAP instance. Due to enhanced security settings, the sapadm user is not authorized any more to retrieve all the instance parameters. Discovery of the SAP instance requires to contact the instance agent using the <sid>adm credentials to retrieve all relevant instance data. Specify the connection data to the SAP Host Agent (IP address of the host), as well as the connection details to the instance agent (Figure 37):
Figure 37 SAP LVM Systems: Find Instances/Hosts The discovery is then started by clicking the Detect Button. The database and SAP instances (central services, application server, diagnostic agent) are identified. Push the Next button to go to the Add instances view. On this screen, the instances are assigned to the appropriate pool. Verify that the Host Name field for each instance matches with the desired virtual name. If an incorrect host name is suggested, select the action Modify Host Names, select the instance to be changed, enter the correct host name in the New Host Name field and click the Modify button to perform the change.
Figure 38 SAP LVM Systems: Add Instances Modify Host Names Select action Auto Assign to New or Existing Systems once the host names are verified. Select all instances and choose a target pool using the selection menu in the Pool for New Systems. Click the Auto Assign button to assign a System name and the pool to each instance. Auto assign will generate a new system name for this system Then click the Next button to go to the Summary screen and Save to save the configuration.
Figure 39 SAP LVM Systems: Add Instances Auto Assign to New or Existing System The base system discovery is now complete. The next steps are to configure additional details for the SAP system, database, and SAP instances. These additional configuration steps are required to enable the system for SAP LVM.
4.7.1.2 System Configuration
Select the new system in the Systems view and switch to Edit mode. On the Basic Configuration screen one can enter a description for the new system, assign a Solution Manager System and enable E-mail and custom notifications.
Page 88 of 156
Figure 40 SAP LVM Systems: Basic Configuration Click the Next button to go to the Provisioning & RFC screen. On this panel, one can enable the flags for various provisioning scenarios. If either one of the copying or renaming flags are enabled, a RFC destination (target client(s) for post copy automation) needs to be defined. The customization panel for a RFC destination will open automatically by checking the Configure RFC Destination flag. Use the Test Connection buttons to verify the data before moving to the next panel with the Next button.
Figure 41 SAP LVM Systems: Provisioning & RFC ACM was not evaluated in the test installation; use the Next button again to go to the next Network Isolation panel. Network Isolation In the network isolation step, all network connections which are still allowed in a Network fenced state are configured. The settings can still be changed during the SAP System Copy/System cloning definition steps. The settings defined here will appear as default allowed connections during the System Copy / System Cloning step (Figure 42).
Page 89 of 156
Figure 42 SAP LVM Systems: Network Isolation In the following example, the network connections to siccserv2 (DNS server for test environment and the ssh protocol were explicitly added as allowed default to the network fencing configuration.
Figure 43 SAP LVM Systems: Network Isolation - Add allowed communications Push the Save button to save the configuration.
4.7.1.3 DB Configuration
The next step is to configure the detail settings for the SAP database instance. Expand the new SAP system in the Systems tab and select the database. Press the Edit button to open the database configuration panel.
Page 90 of 156
Figure 44 Database Configuration: Instance Details Basic Configuration Keep (or change) the settings in the basic configuration view (Figure 45) and then use the Next button to proceed to the Instance properties panel.
Figure 45 Database Configuration: Basic configuration Instance Properties Set the AC-Enabled flag in the Adaptive Enablement section of the screen (Figure 46) and select the appropriate network. In the Requirements section of the screen enter the required SAPS and Memory figures for the system. These attributes will be used during Start- and Relocate actions to determine an appropriate target host candidate for the system. The definition of the target host needs to satisfy the conditions defined in this section. Select the allowed Operating System levels for the system and proceed with the Next button.
Page 91 of 156
Figure 46 Database Configuration: Instance properties Mount Points In this step, the storage configuration for the system is defined. This step depends on the storage provider. For the IBM storage environment, some data needs to be gathered on the host. After un-checking the OS Managed Mounts /Automounter) flag, the screen changes its appearance and one can push the Retrieve Mount List to gather the currently mounted file systems from the host. During a prepare operation the file systems are mounted in the sequence as defined in this panel. Remove the file systems that are not relevant for the database instance and make sure that the file systems are defined in the correct order. Push the Save button to store the database configuration.
Figure 47 Database Configuration: Mount points

4.7.1.4 Configuration of SAP Central Services Instance
After the configuration of the database instance is done, the next step will configure details for the SAP central services instance. Select the Central services instance on the Systems tab and push the Edit button. Then follow the same steps as in the database configuration section to configure the attributes of the central services instance. This time make sure to choose the appropriate file systems required for the central service instance on the Mount points panel.
Page 92 of 156
Figure 48 Central Services: Mount points

4.7.1.5 Configuration of SAP Primary Application Server Instance
Follow the same steps once more to configure the primary application server instance for this system (SAP LVM still uses the term Central Instance to designate the primary application server instance). At this time all configuration steps for a new system are complete and the system should show up as operational under OperationsSystems tab.
Figure 49 Overview of systems and instances 4.8 Additional Configuration Steps
4.8.1 Configuration for SAP System Copy During a SAP System Copy and a SAP System Rename activity SAP LVM starts the SAPInst rename tool on the target LPAR to change the SAP System ID of the system. SAP LVM expects that the SAPInst tool is available in an installation folder accessible on the target LPAR. In the test environment a central NFS server is used as repository, and the directories are mounted on all LPARs. This information is defined in ConfigurationExtended SettingsSystem and Provisioning for each different target OS type. Click the Add button to add a new System Copy definition.
Page 93 of 156
Figure 50 System and AS Provisioning Configuration Select System Copy as provisioning type and proceed with the Next button.
Figure 51 System Copy Configuration: Release configuration type On the next screen enter a descriptive name and select the release version and operating system type.
Figure 52 System Copy Configuration: Basic properties Enter the path to the System Rename tool and proceed to the Summary panel. Verify the configuration on the Summary screen and click Save to store the configuration.
Figure 53 System Copy Configuration: Installation master 4.9 IBM Flex System Manager Configuration IBM Flex System Manager is delivered as a pre-installed appliance node for an IBM PureFlex System. FSM integrates the virtualization management functionality of VMControl, which simplifies the management of virtual resources (server, storage, network, virtual appliance images) and pools of virtual resources.
Page 94 of 156
The following shows the basic steps required to enable VMControl for virtual appliance image management: Discover storage and configure storage system pools Discover servers and configure server system pools Configure fabric management for SAN switches (a deploy operation might have to update the SAN zoning) Discover, request access, and collect full inventories on each object used by VMcontrol (server pools, storage pools, physical server, VIO servers, NIM server, etc) Install the IBM System Director Common Agent, the NIM subagent, and common repository subagent on the NIM server Create a common repository on the NIM server Capture or import a virtual appliance
Please refer to the redbook IBM Systems Director VMControl Implementation Guide on IBM Power Systems (http://www.redbooks.ibm.com/abstracts/sg247829.html?Open) and the VMControl documentation for a detailed description about implementing VMControl and its image deployment capabilities. Once VMControl is configured and the Flex System Manager is defined in SAP LVM, a subset of the available tasks (like capture or deploy an image) can be started directly from the SAP LVM user interface too. 4.9.1 Setup of IBM System Management Security It is possible to use IBM Flex System Manager or IBM Systems Director administrator users for integration with SAP LVM. Some virtualization use cases (OS provisioning and LPAR relocation using Live Partition Mobility) require administration credentials. For virtualization monitoring and management, it is also possible to use credentials that are more restricted. Administrators may allow monitoring and management of particular elements only (e.g. of certain physical servers, LPARs, operating systems, etc.) and thus restrict the view of SAP LVM users. Please check sap note SAP note 1728222 for the latest information about the definition of virtualization manager users and roles. Below we describe how to extend IBM Flex System Manager roles and groups for integration with SAP LVM. 1. Logon to the FSM console and navigate to AdministrationView or modify user accounts. Switch to the Groups tab and use the Create button to create a new group. 2. Go back to the User tab and create a new user. On the User groups panel select the previously created group.
Page 95 of 156
Figure 54 IBM Flex System Manager Security: Group membership 3. Go to AdministrationManage Roles, select the SMUser role and use the Create Like button to create a copy of the role, for example create a new copy called LVM_User. Perform the same step to create a copy of the SMManager role. We named this copy LVM_Manager.
Figure 55 IBM Flex System Manager Security: Create new roles 4. Select the new LVM_User role, click Edit and extend the copied role with following permissions (select the permission under Available permissions and click Add > to move it to Selected permissions): GeneralCreate Group GeneralEdit (Description: Provides an editor for changing an existing group) System Status and HealthMonitors
Page 96 of 156
Figure 56 IBM Flex System Manager Security: Modify role permissions 5. Use the same steps again to extend the copied LVM_Manager role with following permissions: Power On/OffPower Off Now Power On/OffPower On Power On/OffShut Down Power On/OffShut down and power off
6. Navigate to Plug-insDiscovery ManagerResource Explorer and create two static resource groups for monitoring and management. The monitoring group should include physical machines, virtual appliances, storage pools, virtual servers, and all other resources that should be visible in SAP LVM. The management group should include only the virtual servers that will be manageable through SAP LVM.
Page 97 of 156
Figure 57 IBM Flex System Manager Security: Define static groups 7. Assign the new roles and resource groups directly to the user or the user group created in step 1 and 2. In this example, we use the user group. Navigate back to AdministrationView or modify user accounts. Switch to the Groups tab, select the SAP LVM group and click Assign Role. Select the LVM_User role under User Role and the LVM_Monitoring group under Selected resource groups. Click the Add > button to add this combination to the Assigned Roles. Perform the same step with the copy of the SMManagement role and the LVM_Management group.
Figure 58 IBM Flex System Manager Security: Assign groups and roles to user group The configured user is now available for the integrating IBM Flex System Manager as a virtualization manager in SAP LVM.
Page 98 of 156
5 Detailed Description / Execution of Use Cases

5.1 LVM Standard Edition Use Cases This section describes the use cases included in the standard version of SAP LVM. Basically the SAP LVM Standard Edition allows to start, stop, and relocate SAP Systems. 5.1.1 Description of the LVM Standard Edition Use Cases Following use cases are possible: Prepare/unprepare SAP instances and systems Start/stop SAP instances and systems Relocate SAP instances and systems Start/stop hosts Monitoring
5.1.1.1 Prepare/Unprepare SAP Instances and Systems
The Standard Edition of SAP LVM is able to prepare and unprepare SAP systems and instances. In case of a prepare operation, the SAP instance or system is made ready to be started on a host. During the prepare phase, the storage volumes belonging to an SAP system are mapped to the LPAR, the volume groups are imported, the file systems are mounted, the network isolation on the LPAR is being setup, and the virtual hostnames are assigned to a network interface. During an unprepared operation, file systems volumes belonging to a SAP system are unmounted, the volume groups are exported, the storage volumes are unmapped, and the network isolation and virtual hostnames are being removed. Prerequisite is that the SAP instance or system to be unprepared is stopped. This allows to map and start the SAP system to/on the same host, or to/on another host. The prepare/unprepare can be run on a SAP instance or on a complete SAP system including all instances belonging to an SAP system. If the complete SAP system is affected, then the function is called mass operation. Prepare/unprepare triggers following calls: Storage CLI commands in order to attach or detach storage volumes containing SAP systems to / from certain OS hosts OS specific storage libraries for handling the storage mapping OS specific libraries for handling the SAP instance service OS specific libraries for handling the network setup
5.1.1.2 Start/Stop SAP Systems and Instances
In order to start a SAP instance or system, it needs to be successfully prepared. After a SAP instance or system has been stopped, it can be unprepared. The prepare/start and stop/unprepare tasks can be executed as a single workflows. As for prepare/unprepare mass operations allow to start/stop complete SAP systems versus single SAP instances. Start / stop triggers following calls: OS specific libraries for handling the SAP tasks start / stop
5.1.1.3 Relocate SAP Instances and Systems
The relocation of SAP instances and systems allows to move SAP instances or systems to another host. There are two flavors of SAP instance or system relocation:
Page 99 of 156
IBM PowerVM based relocation The complete LPAR including the SAP system and operating system is moved (also called migrated) from one physical host to another physical host. In that case, LVM uses IBM PowerVM Live Partition Mobility (LPM). The LPAR and SAP system stay up and running during the execution of LPM. LPM is used e.g. for hardware maintenance activities. Hypervisor based relocation triggers following calls: o IBM PowerVM specific libraries for handling Live Partition Mobility in the HMC or Systems Director SAP LVM relocation This means that the SAP system is moved from one managed host to another managed host. This causes a short SAP downtime as the SAP system needs to be stopped on the source host and started on the target host. The process automates following functions: o Stop SAP instance or system o Unprepare SAP instance or system o Prepare SAP instance or system on the new LPAR o Start SAP instance or system on the new LPAR As for prepare/unprepare or start/stop, mass operations allow to relocate complete SAP systems versus single SAP instances. SAP based relocation can be used for operating system maintenance. SAP based relocation triggers following calls: o Storage CLI commands in order to attach or detach storage volumes containing SAP systems to/from certain OS hosts o OS specific storage libraries for handling the storage mapping o OS specific libraries for handling the SAP instance service
Figure 59 and Figure 60 illustrate the two relocation scenarios.
Page 100 of 156
Figure 59 IBM PowerVM Relocate
Figure 60 SAP LVM Relocate

Page 101 of 156
IBM SAP International Competence Center 5.1.1.4 Start/Stop Hosts
With SAP LVM hosts can be started or stopped. Caution: Powering off hosts without stopping and unpreparing the SAP systems running inside the LPAR may cause data loss or data corruption. Starting or stopping LPARs triggers following calls: IBM PowerVM specific libraries for handling Live Partition Mobility in the HMC or Systems Director
5.1.1.5 Monitoring
SAP LVM provides several types and levels of monitoring, e.g. CPU consumption, memory consumption, dashboard, history data etc. SAP LVM communicates with the virtualization manager to retrieve monitoring data about IBM Power physical servers and logical partitions. Optionally, SAP LVM tries to gather CPU utilization data for discovered elements (physical servers and LPARs). SAP Notes 1728222 (for IBM Systems Director VMControl adapter) and 1728293 (for IBM HMC adapter) provide configuration details of components involved in this scenario. Following setups are supported: Monitoring through IBM Systems Director VMControl adapter Monitoring through HMC adapter 5.1.2 Executing the LVM Standard Edition Use Cases
5.1.2.1 Prepare/Unprepare SAP Instances and Systems
The following section describes the required steps to prepare a SAP instance. As the unprepare process is very similar, it will not be shown separately. As a prerequisite, the SAP system needs to be installed and discovered in SAP LVM with correct storage volumes and mount points. To prepare a SAP instance: Log in to SAP LVM and navigate to OperationsSystems. Select the pool in which the SAP system resides, then the SAP system you want to prepare. Select the instance and assign a host in the Host Properties field. Click the small arrow in the right lower corner of the button Prepare and Stop button and select Prepare.
Page 102 of 156
Figure 61 Prepare SAP Instance You can monitor the progress in the Logs tab (see Figure 63).
5.1.2.2 Start/Stop SAP Systems and Instances
The following section describes the required steps to start a complete SAP system including the prepare step. As the complete SAP system is started, this is a mass operation. The stopping process is very similar, it will not be shown separately. As a prerequisite, the SAP system needs to be installed and discovered in SAP LVM with correct storage volumes and mount points. To start a complete SAP instance including preparation: Log in to SAP LVM and navigate to OperationsSystems. Select the pool in which the SAP system resides, then the SAP system you want to start. Select or enter a host for each instance to be started and select the operation Mass Start (including prepare if possible). Click the Execute button to start the operation.
Page 103 of 156
Figure 62 Start SAP system: Mass Operations The logs for an operation are available under MonitoringLogs:
Figure 63 Logs of a 'Start SAP system' Operation
Page 104 of 156
IBM SAP International Competence Center 5.1.2.3 Relocate SAP Instances and Systems
The following section describes the required steps to relocate a complete SAP system using SAP LVM relocation. That is, only the SAP system (not the host) is moved to another LPAR. As the complete SAP system is relocated, this is a mass operation. As a prerequisite, the SAP system needs to be installed and discovered in SAP LVM with correct storage volumes and mount points, and it needs to be running. To start the relocation of a complete SAP system: Log in to SAP LVM and navigate to OperationsSystems. Select the pool in which the SAP system resides, then the SAP system you want to relocate. Select the instances and select a target host for each instance. Select the operation Select Mass Relocate All in State Running and click the Execute button.
Figure 64 Relocate SAP System: Mass Operation
Page 105 of 156
IBM SAP International Competence Center 5.1.2.4 Start/Stop Hosts
The following section describes the required steps to stop a LPAR. As the start process is very similar, it will not be shown separately. Make sure that all SAP systems and instances are stopped and unprepared. Otherwise, you risk a data loss or a database corruption. There are two options to stop a LPAR: 1. OS Shutdown: LVM triggers an operating system shutdown 2. Power Off: LVM just switches off the LPAR Using the OS Shutdown is recommended. Use Power Off only in case the shutdown is no longer working, e.g. the LPAR is not reacting to any commands. Power Off will not close network connections, unmounts file system etc. To shut down a host: Log in to SAP LVM and navigate to OperationsHosts. Select the pool in which the host resides, then the host you want to stop. Select the Virtual Host tab, then Deactivate and select Deactivate (OS Shutdown).
Figure 65 Shutdown Host
Page 106 of 156
IBM SAP International Competence Center 5.1.2.5 Monitoring
The following section describes how to access the different monitoring functions. Dashboard The Dashboard is the first screen opened per default when you log in to LVM. It provides information e.g. regarding SAP systems, physical resources, and virtual resources. By selecting Add Pool you can open a window showing an additional subset of information.
Figure 66 SAP LVM Dashboard Visualization The Visualization tab provides a visual view on the infrastructure and SAP systems and instances running on the infrastructure.
Page 107 of 156
Figure 67 SAP LVM Infrastructure Visualization Performance monitoring Performance monitoring enables you to display metrics for specific entities (such as SAP systems, specific instances, or virtual machines) for a specific time period. The historical monitoring data is fed to automatic capacity management (ACM) to provide response time data. To open the performance monitor: Log in to SAP LVM and navigate to MonitoringPerformance. Select the pool in which the host resides, then the host you want to stop. Select the Virtual Host tab, then Deactivate and select Deactivate (OS Shutdown). Select one or more entities on the left (for example, application server) and click the metric you want to display (e.g CPU usage) and the period (e.g. hour).
To add entities to the display of a metric, specify the entity in the Entity field (or choose the entity using the value selector) and choose Add.
Page 108 of 156
To add the metric to the dashboard as a pod, click the Add to Dashboard button. If you have added entities to the display of the metric, the metric is displayed on the dashboard only for the first entity.
Figure 68 SAP LVM Performance Monitoring
5.2 SAP LVM Enterprise Edition Use Cases The execution of SAP LVM Enterprise edition use cases require that LVM Enterprise Edition (VCM_ENT.SCA) is deployed to the managing SAP NetWeaver Java Web AS system. 5.2.1 SAP System Copy Use Cases The SAP LVM Enterprise Edition Use-Cases include the SAP System Copy use cases Figure 69 illustrates the three different scenarios for SAP System copies in SAP LVM: SAP System Clone Initial SAP System Copy SAP System Copy Refresh
Page 109 of 156
Figure 69 SAP System Copy Scenarios Supported by SAP LVM (Source SAP)
5.2.1.1
SAP System Clone
In the SAP System Clone scenario, an exact duplicate of an existing SAP system is created. Content and system configuration are 100% identical in all aspects to the original system. To avoid conflicts, e.g. with interfaces dealing with the original system landscape, the clone SAP system is fully network isolated from the rest of the environment before the SAP target system can be started. Therefore, the clone system is established in an isolated networking environment. AIX IPsec allows restricting the outbound TCP/IP network communication to hosts/ports which dont participate in any interface communication of the SAP system.
Page 110 of 156
SAP System cloning is realized in two different versions: Storage based System Clone For storage based cloning the system to be cloned must be adaptive enabled: This includes o virtual IP addresses for both database and central instance o Storage Setup separating all the SAP and database entities from the operating system image. The SAP and database entities must be deployed to storage supported by the SAP LVM storage adapter. The target host must match all prerequisites for a relocation target of adaptively installed systems (e.g. users, services, etc). Virtualization based For virtualization based System Cloning the system to be cloned must be installed in the rootvg of a LPAR that is managed by the SAP LVM virtualization manager, and which is registered with SAP LVM. In addition, virtualization based cloning is supported with an offline database copy only. During virtualization based System Cloning, the SAP System is stopped, the LPAR is captured via IBM Systems Director/ VMControl, and is deployed to the new target LPAR afterwards.
Initial SAP System Copy
5.2.1.2
During a SAP System Copy a duplicate of an existing SAP system is created with a different system name and a unique SAP system ID (SID). The target system is updated with the new system information, and can run in the same network environment as original/source SAP system. As the first step, the system copy involves the creation of a system clone to create the initial target system. Then this initial target system is modified during subsequent steps to update the settings according to the new SAP System ID. A special version of SAPInst, SAPInst Rename is used to change the SAP system ID during this process. Additional post configuration tasks in the SAP System are achieved via the Post Copy Automation (PCA) tool. PCA provides task lists, with a predefined sequence of configuration tasks to configure the extensive technical scenarios automatically. In the SAP LVM environment the execution of a PCA task list is embedded in the overall system copy workflow.
5.2.1.3 SAP System Refresh
Given are two systems 'SAP source' and 'SAP target' which exist and are configured in a copy-relationship (target is copied from source). SAP LVM then allows to 'refresh' the 'target' system by: exporting configuration data from the 'target' system SAP System Copy for copying the source system to the 'target' system (same target SID, same hostnames as previous system copy) importing the formerly exported configuration data into the new 'target' system The System Refresh feature has the same requirements as the System Copy.
5.2.1.4 SAP System Rename
The SAP System Copy can be executed in two steps also: Create a SAP System Clone in a first step Then use this clone later on, and rename it to a new SAP System ID. Like in scenario 5.2.1.2, the SAPInst Rename tool is invoked for the SAP System ID change.
At first, the SAP system to be renamed needs to be enabled for the SAP System Rename by executing following steps: Go to ConfigurationSystem. Select the new cloned System and click the Edit button. In the configuration screen, activate the checkbox Renaming and click Save.
Figure 70 Enable SAP System for SAP System Rename 5.2.2 SAP ACM (Automatic Capacity Management) Use Cases The SAP ACM use cases are SAP LVM enterprise use cases too. However, they will not be discussed in further detail here. Just note that the following considerations need to be taken into account for the ACM Use-Cases: Both adaptively installed SAP systems and traditionally installed SAP systems are supported for ACM One dialog instance per host is supported All ACM managed dialog instances must be in the same set of logon groups reserved for use by ACM. SAP CCMS must be installed and operational for each managed SAP system. For ACM-managed VMs in the same LVM pool, all the OS users must be configured identical and must be defined in the central user store (LDAP).
Page 112 of 156
5.2.3 Required Preparation Steps Table 17 includes an overview about the required initial setup step before a system can be cloned, copied, refreshed or renamed with SAP LVM. For each of the required steps it references to the section in this document where the step is described in more detail.
Setup Procedure in SAP LVM Configuring Networks Configuring Storage Providers Configuring User Management Configuring Name Servers Configuring Remote Function Call Destinations Setting the Infrastructure Assignment Configuring System Rename Configurations Configuring Mount Points Configuring Java Post Copy Automation Configuring a universally unique identifier (UUID) Enabling the system for (See chapter 5.2.1.4, Figure 70) Clone System Copy System Refresh System/ Database Rename System
Required (See chapter 4.5) Required, if the source system is adaptively installed. Not Required (See chapter 4.3) Not part of the test environment: All users for the target systems had been pre-defined in LDAP Not part of the test environment: All virtual hostnames and IP addresses had been pre-defined in DNS. (See chapter 5.2.4.1 for additional information) Not Required Required (See chapter 5.2.5.2) Required (See chapter 4.7.1.1 and 4.7.1.2) Not Required Required (See chapter 4.8.1)
Required, if the source system is adaptively installed (See chapter 4.7.1.3 and 4.7.1.4) Not Required Not part of the test environment: SAP ABAP Stack tests only Required (See chapter 3.4.5.4) Not Required
cloning
copying
refreshing
renaming
Table 17 Required Setup Procedures 5.2.4 Execution of a SAP System Clone Operation The following section describes the required steps for starting the SAP system clone in SAP LVM. As a prerequisite, a SAP source system need to be installed and discovered in SAP LVM, and need to be enabled for SAP System cloning in the SAP LVM configuration A SAP System Clone is started on the System and AS Provisioning tab in the Provisioning view. Select the SAP System to be cloned and click the Clone System button.
Page 113 of 156
Figure 71 Start SAP System Clone SAP LVM starts the System Cloning dialog. In the first screen, the LVM Pool is shown, and a short name and a description for the target system need to be specified. Accept the default settings, and click Next to go to the next screen.
Figure 72 SAP System Clone: Basic Data of Target System In the next screen, target hosts for the SAP system clone are specified: 1. In this example the default value Use Existing Host was chosen for the Central Services instance and the Central instance. SAP LVM takes an existing LPAR out of the pool of hosts. The instances will be cloned and attached to this already existing host. For the Database instance, Provision of New Host is selected. Here a new LPAR will be created as the target host for the SAP System Clone. Once you click Add, SAP LVM calls the virtualization manager (in our case IBM Flex System Manager VMControl) to deploy the new target host as part of the cloning process. 2. Click Next to continue with the next selection screen
Page 114 of 156
Figure 73 SAP System Clone: Host Selection of Target System In the next screen, the virtual host names and the network for the SAP system clone are defined. The virtual hostname per default is derived from naming conventions. The short name specified in Figure 72 is inserted into the virtual hostname.
5.2.4.1 Excursus: DNS Name Resolution
DNS Name Server Considerations
In our test landscape all target IP addresses for the clone IP labels are predefined in DNS by an appropriate address range.
Role SAP Source SAP Target IP Label (DNS server) cl6ci01 cl6db01 cl6cloneci01 cl6clonedb01 IP Address 192.168.165.116 192.168.165.126 192.168.165.136 192.168.165.146
Table 18 Examples for IP addresses SAP Note 1572841 - Setup of name server update of SAP LVM describes an alternative solution where SAP LVM automatically updates/creates virtual hostnames in the DNS server. SAP LVM then creates the respective A (forward) and PTR (reverse) lookup records for IPv4 addresses in the DNS server at the runtime during the SAP System Clone operation. During a SAP System Destroy operation these records previously added are removed. This mechanism ensures that in the DNS server the new IP addresses are assigned to the new IP label. In the test environment however all virtual hostnames and IP addresses had been pre-defined in DNS and the automatic name server update was not tested.
DNS Behaviour on the Target Host
On the target host, things may look slightly different. SAP LVM acquires the new IP addresses on the target host during the prepare operation, e.g. On the target host:
root> Name en0 en0 en0 en0 netstat -in Mtu Network 1500 link#2 1500 9.153.164 1500 192.168.164 1500 192.168.164 Address 86.1f.b8.54.6b.2 9.153.164.137 192.168.165.146 192.168.165.136
Page 115 of 156
However, the name resolution on the target host still states the source IP labels. This is required for SAP system operation of the clone. The clone system is a 100% identical clone of the source system and the virtual IP labels need to be unchanged on the target host (as e.g. the virtual IP labels appear in the instance profiles, start scripts, etc.). On the target host:
root> Name en0 en0 en0 en0 netstat -in Mtu Network 1500 link#2 1500 9.153.164 1500 192.168.164 1500 192.168.164 Address 86.1f.b8.54.6b.2 siccp137 cl6db01 cl6ci01
To achieve this behaviour, SAP LVM adds new entries to the local /etc/hosts file: On the target host:
root> cat /etc/hosts 192.168.165.136 cl6ci01.isicc.de.ibm.com cl6ci01 #SAP-LM: Added to modify local hostname resolution for cloning DO NOT MODIFY 192.168.165.146 cl6db01.isicc.de.ibm.com cl6db01 #SAP-LM: Added to modify local hostname resolution for cloning DO NOT MODIFY
With local name resolution, the new target IP addresses are mapped to the source IP labels. In order to ensure that the mechanism works, it is mandatory that the DNS search order evaluates the local /etc/hosts first. This ensures that the local host file supersedes the (external) nameserver. The nameserver gets contacted only in case the record to be resolved is not found in the local hosts file.
root> cat /etc/netsvc.conf # order of search: 1. /etc/hosts 2. nameserver hosts = local,bind4
Back to the selection screen of the SAP system clones, see Figure 74. 1. The network name needs to be selected 2. Click Next to continue with the next screen
Page 116 of 156
Figure 74 SAP System Clone: Virtual Host Names and Networks In the next screen, all the storage definitions are made for the SAP system clone. 1. SAP LVM provides proposals based on built in naming conventions for the new volume names on the storage system, and logical volume names on the operating system. You can accept the defaults, or overwrite the values based on your own naming conventions. 2. Selection of Full Clone allows to create a new, independent set of target volumes (which no longer depend on the source volume) 3. SAP LVM also allows specifying the storage pool on the storage systems which should be used the hold the target system. If you dont specify it, then LVM will use the same storage pool where the source system is located. 4. Click Next to get to the next selection screen.
Figure 75 SAP System Clone: Storage Volumes

On the next screen, specify the database consistency method. By selecting Online Copy/Suspend IO Mode, SAP LVM will place the DB2 UDB database into write suspend state before the FlashCopy/SnapShot is triggered in the storage system. When the FlashCopy/SnapShot is finished, SAP LVM will initiate a write resume to bring back the database into the operational state.
Figure 76 SAP System Clone: Database consistency In the next screen, all the port definitions for the network fencing of the SAP system clone are specified. The highlighted item under allowed connections is the DNS name server. In our test environment, the standard /etc/hosts file didnt include entries for the virtual IP labels. All the name resolution is done via DNS to the DNS server. So it is important that the LPAR is able to communicate with the DNS server in a fenced state too. Therefore the DNS server (here: siccserv2) was added as allowed communication already during the SAP system setup in SAP LVM. Click the Next button to get to the final confirmation screen.
Figure 77 SAP System Clone: Define Allowed Outgoing Connections for System Isolation
Page 118 of 156
The final confirmation screen appears. It is possible to review all the parameters defined in the steps before. Click Start System Cloning to finally start SAP System Clone execution.
Figure 78 SAP System Clone: Summary After starting the SAP System Cloning process, the Monitoring screen is launched and the progress of the SAP System Clone can be tracked. 5.2.5 Execution of a SAP System Copy The following section describes the required steps for starting the SAP system copy in SAP LVM. As a prerequisite, a SAP source system needs to be available and discovered in SAP LVM, and need to be enabled for SAP system copies in the SAP LVM configuration.
5.2.5.1 Preparations for Post Copy Automation (PCA)
Post Copy Automation (PCA) is part of SAP NetWeaver Landscape Virtualization Management, enterprise edition solution. This section illustrates how to set-up PCA initially. Several preparation steps before set-up of SAP Post Copy Automation need to be executed in the source system first: Details are described in the document "ABAP Post Copy Automation Guide for Enterprise Edition". See SAP NetWeaver Landscape Virtualization Management Installation & Upgrade GuidesSAP NetWeaver Landscape Virtualization Management 2.0 and in SAP Note 1614266 - System Copy: Post Copy Automation (PCA) / LVM for further details.
Page 119 of 156
The file "Note_Overview.pdf" (which is attached to SAP Note 1614266) lists a set of additional SAP notes which need to be implemented as a prerequisite for PCA in the source system. Dependent on the actual support package stack of the SAP system they may be already included in the system, and corrections may need to be applied to the SAP system (via transaction SNOTES). The ABAP program Z_SAP_NOTE_ANALYZER (which is also attached to SAP Note 1614266) automates this procedure. The SAP system is checked, and according to its actual support package stack the required SAP notes content is downloaded directly during execution of the from Z_SAP_NOTE_ANALYZER report, the content is transferred into the SNOTES notes assistant transaction and added to the queue. If all the pre-requisites are met, then PCA is installed via the Post Copy Automation Installer (PCAI) automatically into the managed source system. See also the following notes and the referenced notes that are mentioned in there: Note 1589145 - Task Manager for Technical Configuration, Note 1589175 - System Copy: Task Content for Task Manager Further details are described in chapter 3 and chapter 4 of the ABAP Post Copy Automation Guide for Enterprise Edition. PCA is part of the enterprise edition of the SAP NetWeaver Landscape Virtualization Management solution. PCA is shipped with SAP NetWeaver packages, but is deactivated per default. The License Enabler add-on PCAI_ENT 1.0 activates PCA and is required in the system to enable the execution of PCA task lists. Install the add-on PCAI_ENT 1.0 before the first system copy of the system using transaction SAINT. Details are described in chapter 5 of the ABAP Post Copy Automation Guide for Enterprise Edition. To grant the privileges to execute PCA activities for the administrator (dialog user) the corresponding roles SAP_BC_STC need to be assigned to all relevant users. This includes the system administrator user responsible for PCA activities, and the technical users that will be used via remote access by SAP LVM. Details are described in chapter 7 of the ABAP Post Copy Automation Guide for Enterprise Edition. PCA provides task lists, with a predefined sequence of configuration tasks to configure extensive technical scenarios automatically. To prepare the run of these task lists, the ABAPbased task manager for technical configuration is used. In the SAP LVM environment, the execution of such a task list is induced without calling the task manager for technical configuration directly. Further details are described in chapter 8 and chapter 9 of the ABAP Post Copy Automation Guide for Enterprise Edition.
5.2.5.2 Add the RFC destinations for PCA to the SAP System configuration
In the systems configuration, define the RFC destinations that will be used for the PCA execution for the SAP system (see Figure 79). Define the logon parameters for the main destinations, and click Test Connection. An attempt to connect to the SAP system is made. Press Retrieve Clients and the list will be populated with all clients of the existing system. Different PCA activities may be executed specific for each client of the SAP system. Adjust the Users accordingly, and define the passwords.
Page 120 of 156
Figure 79 Define RFC Destination for PCA Execution

5.2.5.3 Start the SAP System Copy
A SAP System Copy is started in the System and AS Provisioning tab in the Provisioning view. Select the SAP system to be copied and click the Copy System button.
Figure 80 Start SAP System Copy
Page 121 of 156
On the next screen, enter the SAP system ID and master password for the target system. Both will be used later during the SAPINST_RENAME step to adapt the configuration. The additional description may be adjusted additionally. This will be the description for the new system in the SAP LVM configuration. Click the Next button to go to the next step.
Figure 81 SAP System Copy: Basic Data of Target System In the second step the target host(s) for the system copy is defined: 1. In this example the default value Use Existing Host was chosen. SAP LVM takes an existing host out of the pool of hosts. The instances will be cloned and attached to this already existing host. 2. Alternatively, you may select Provision of New Host for one or more instances. In this case, a new LPAR would be created as the target host for the SAP instance(s). SAP LVM would call the virtualization manager, here IBM Flex System Manager VMControl, to deploy the new target host as part of the SAP system copy process. 3. Click the Next button to continue with the next selection screen
Figure 82 SAP System Copy: Host Selection of Target System
In the next step, the network configuration for the target system is specified.
The IP addresses for the target system are proposed according to the naming conventions for the virtual IP labels (<sid>pasXX, (<sid>scsXX, <sid>dbsXX). The addresses and labels are either predefined in the DNS (respectively in the local host files of the SAP LVM system and the involved resources) already. Otherwise, SAP LVM may directly update the DNS domain name server for the new entries, however that need to be defined and configured according to SAP Note 1572841 - Setup of name server update of SAP LVM first. A network name for the Network needs to be selected by using the drop-down list. Click Next to continue with the next step.
Figure 83 SAP System Copy: Virtual Host Names and Networks In Step 4, the Instance number(s) for the Instances of the target system are defined. Per default, the target system will be installed using the same instance numbers as the source system. Click Next to continue.
Figure 84 SAP System Copy: SAP Instance Numbers In step 5 the storage configuration for the target system is defined. The storage image for the target system is created as a SnapShot/FlashCopy of all source volumes to the target volumes. The main structure like the overall number of AIX volume groups and their content cannot be changed. Names for volume groups, logical volumes and filesystem names can be adjusted however. The storage pool for the target volumes can be specified.
Page 123 of 156
Per default, all SAP System IDs in the filesystem names are updated to the new SAP System ID of the target system.
Figure 85 SAP System Copy: Storage Volumes In case the storage connection to the IBM SVC is configured to use ssh, a warning message is shown, as the recommendation is to use SMI-S. Click Ignore warnings for the step and continue with Next button.
Figure 86 SAP System Copy: SVC CLI Warning Message In the next screen, specify the database consistency method. By selecting Online Copy/Suspend IO Mode, SAP LVM will place the DB2 UDB database into write suspend state before the FlashCopy/SnapShot is triggered in the storage system. When the FlashCopy/SnapShot is finished, SAP LVM will initiate a write resume to bring back the database into operational state.
Page 124 of 156
Figure 87 SAP System Copy: Database consistency In step 7 the OS administration users are specified for the target SAP system. If LDAP is used for central user management, then the users need to be created manually before the SAP system Copy is started. sapinst is currently not able to handle LDAP users correctly. However, SAP LVM is able to create local users as part of the SAPInst Rename step. Click Next to continue the next step.
Figure 88 SAP System Copy: User and Group Management In step 8 further additional settings for DB2 schema user and DB2 instance are specified for the new SAP System ID. For DB2 UDB, the schema user cannot be changed during the process and must be set identical to the schema user of the source database. The name of the database instance for the target system is derived from the SAP System ID according to SAP naming conventions. The DB2 Security Administrator can be replaced (from db2<src-sid> to db2<tgt-sid>), which will cause a re-assignment of all DB2 objects to the new security admin.
Click Next to continue.
Figure 89 SAP System Copy: SAPinst Release Configuration In step 9, the allowed ports/connections for the network fencing step are defined. Additional ports/hosts allowed for communication can be selected out of the set of Current connections on host from the source system, and then added to the allowed outgoing connections on the target hosts. To avoid additional manual steps it is best-practise to include all the necessary hosts/ports required for outbound connections of the system already in the SAP system definition. Click Next to go to the next step.
Figure 90 SAP System Copy: Define Allowed Outgoing Connections for System Isolation In step 10, the SAP clients for executing the Post-Copy Automation (PCA) steps can be defined.
Page 126 of 156
Figure 91 SAP System Copy: Post Copy Automation All available tasks can be gathered from the source system, and can be selected accordingly. Click the Next button to continue to the next selection screen.
Figure 92 SAP System Copy: Select Task List Before actually starting the SAP System Copy, a summary screen is displayed. All the parameter settings can be reviewed by expanding the different subsections, and eventually adjust them once again.
Page 127 of 156
Figure 93 SAP System Copy: Summary View Click Start System Copy to finally start the SAP System Copy. The Monitoring screen is launched and the progress of the system copy can be tracked. 5.2.6 Execution of SAP System Refresh Prerequisite for the SAP System Refresh is a SAP System Copy run that was previously completed successfully. The execution of the SAP System Rename is very similar to the steps before. Start with the Provisioning view. Click the Refresh System button to start the parameter dialog.
Page 128 of 156
Figure 94 SAP System Refresh: Start System Refresh In the first screen base configuration parameters can be adjusted. 1. Specify and confirm the Master Password 2. Click Next to continue with the next selection screen
Figure 95 SAP System Refresh: Basic Data of System to be Refreshed Acknowledge the target host selection in the following screen and click Next again.
Page 129 of 156
Figure 96 SAP System Refresh: Host Selection of Target System Acknowledge the virtual hostnames and network settings and click Next.
Figure 97 SAP System Refresh: Virtual Host Names and Networks In the next screen, all the storage definitions are made for the SAP system clone: 1. SAP LVM provides proposals based on built in naming conventions for the new Volume names on the storage system, and Logical Volume names on the operating system. You can accept the defaults, or overwrite the values based on your own naming conventions. 2. Selection of Full Clone allows to create a new, independent set of target volumes (which no longer depend on the source volume) 3. SAP LVM also allows specifying the storage pool on the storage systems which should be used the hold the target system. If you dont specify it, then LVM will use the same storage pool where the source system is located. 4. Click Next to get to the next selection screen.
Page 130 of 156
Figure 98 SAP System Refresh: Storage Volumes In the next screen, specify the database consistency method. By selecting Online Copy/Suspend IO Mode, SAP LVM will place the DB2 UDB database into write suspend state before the FlashCopy/SnapShot is triggered in the storage system. When the FlashCopy/SnapShot is finished, SAP LVM will initiate a write resume to bring back the database into operational state.
Figure 99 SAP System Refresh: Database consistency
Page 131 of 156
In the next step, acknowledge the user settings. As it is a SAP system refresh, the users should already exist. Click Next to continue.
Figure 100 SAP System Refresh: User and Group Management In the next screen, enter the password for the schema user and click the Next button to proceed to the next step.
Figure 101 SAP System Refresh: SAPinst Release Configuration In the next screen, acknowledge or adjust the isolation settings during the network fencing. Click Next to continue.
Page 132 of 156
Figure 102 SAP System Refresh: Define Allowed Outgoing Connection for System Isolation Adjust or confirm the PCA settings and click Next to continue.
Figure 103 SAP System Refresh: Post Copy Automation In the next screen, select whether to delete storage volumes at the end of the SAP System refresh. Chose the default value to not delete volumes, and click Next to get to the next selection screen.
Page 133 of 156
Figure 104 SAP System Refresh: Storage Volumes Finally review the settings and start the operation by clicking the Start System Refresh button.
Page 134 of 156
Figure 105 SAP System Refresh: Summary 5.2.7 Execution of SAP System Rename The execution of the SAP System Rename is very similar to the steps before. Start with the Provisioning view. The system needs to be enabled for SAP System Rename. Only then the appropriate button is shown. Click Rename system to start the parameter dialog.
Page 135 of 156
Figure 106 Start SAP System Rename In the following screen, enter the basic data for the SAP System Rename. 1. Enter and confirm the master password 2. Click Next to get to the next selection screen
Figure 107 SAP System Rename: Basic Data of Target System For convenience, the virtual hostname can be changed on the next screen, as the one from the system clone will not fit to the naming conventions. 1. Then, select a network 2. Click Next to proceed
Page 136 of 156
Figure 108 SAP System Rename: Virtual Hostnames and Networks SAP Instance number for the renamed system can be specified or, per default, use the same as the source system. Click Next to continue.
Figure 109 SAP System Rename: SAP Instance numbers On the next panel, the mount points are adjusted to the new SAP System ID. In most cases you can accept the default names, except if you specifically want to update certain mount points. Click Next to proceed.
Page 137 of 156
Figure 110 SAP System Rename: Mount Data Select the administrative users for the new SAP System. If you use LDAP, create users and groups manually in LDAP before, as SAPinst is currently not able to create LDAP users correctly. However, SAPInst Rename is able to create the SAP users locally in AIX. If the users already exist in LDAP or locally in the operating system, then the checkbox User already exists is automatically flagged. Click Next to proceed to the following screen.
Page 138 of 156
Figure 111 SAP System Rename: User and Group Management In the next step, the settings for the database connect user are specified. For DB2, the connect user cannot be changed during the System Rename. 1. Specify the password for the connect user 2. Click Next to proceed
Figure 112 SAP System Rename: SAPinst Release Configuration
Page 139 of 156
The next screen describes the allowed communications during the network fencing phase. Ensure that all necessary connections (e.g. to the DNS nameserver) are allowed, but that all potential dangerous communication is prohibited. Click Next to proceed to the following screen.
Figure 113 SAP System Rename: Define Allowed Outgoing Connections for System Isolation In the next screen, the SAP target clients for Post Copy Automation execution are specified. In case you want to disable PCA, remove all the potential clients from the list. If the table is empty, then the PCA wont be invoked for any client. Click Next to proceed to the final confirmation screen.
Figure 114 SAP System Rename: Post Copy Automation
Page 140 of 156
In the final selection screen, the parameters can be review before the actual start. Then click the Start System Renaming button to start the execution.
Figure 115 SAP System Rename: Summary
Page 141 of 156
5.2.8 Additional Considerations for SAPInst Rename See also SAP Note 1619720 - System Rename for SAP Systems based on SAP NetWeaver for additional information. In our lab scenario we identified a few issues in the context of SAP system copies. (Some or all of them may be already fixed with the latest versions of SAP System Rename/SAP Kernel, but this was not re-tested). 1. The SAP System Copy in SAP LVM procedure doesnt handle DB2 UDB with Automatic Storage Management (ASM) correctly. o During the SAP System Copy, the SAP Kernel executable brdb6brt is called on the source system to generate a parameter file relocate.template. The content of the file is later used on the target system during the relocatedb step: The db2relocatedb tool allows to move the location of an entire DB2 UDB database to the new file system structure (e.g. change of <Source SID> to <Target SID> in both database and filesystem structure). In case of ASM, additional parameters are required in the template file: 1. STORAGE_PATH 2. CONT_PATH Those parameters were not provided from brdb6brt o The ASM directory structure follows the conventions /db2/<SID>/sapdataX/db2<sid>/NODE0000/<SID> Not all the <SID> and <sid> entries are adapted by SAP LVM to the new target <SID> / <sid> values of the target system Resolution: Handled in Event script, see below 2. SAPinst and SAP LVM dont handle user creation in LDAP correctly Make sure that the ABAP Connect User on the target host has the following group sets assigned groups=db<src-sid>mon,db<srcsid>mnt,db<tgt-sid>mon,db<tgt-sid>mnt Resolution: Handled in Event script, see below 3. SAPSYSTEM RENAME didnt handle the update to db2nodes.cfg if a full qualified hostname is set as hostname on the target host If hostname and netname are specified in db2nodes.cfg, only the hostname is changed. If only the hostname is specified, then it is translated erroneously Resolution: Handled in Event script, see below 4. An issue with db2chgpath was identified in the test environment (DB2 UDB V9.7 FP5SAP) Resolution: Fixed in DB2 UDB V9.7 FP6 (APAR IC80849).
Page 142 of 156
5. Update to AIX 6.1 TL7 SP4 (6100-07-04-1216) introduced a new issue IV22062 on some of the nodes. Resolution: Install eFIX on top of AIX 6.1 TL7 SP4 root> emgr -l ID STATE LABEL INSTALL TIME UPDATED BY ABSTRACT ====================================================== 1 S IV22062s04 06/11/12 12:42:43 Ifix forIV22062@6.1TL7SP4 Item 1.), 2.) and 3.) are addressed by a Pre-Event script, which is invoked before SAPInst rename is started: This is realized via a Pre-Hook in SAP LVM. The following outlines how to implement such a hook. See also SAP Note 1465491 - Provider Implementation Definition for further details.
5.2.8.1 Create the Script and Register it with the Host Agent
The script is invoked for both SAP Central Instance and Database. It checks, if it is running for the database, and then identifies the Source SID based on the environment. Target SID is specified as argument. The script provides the following functionality Check and eventually create/correct the LDAP users and groups required for the target system Handle DB2 UDB ASM o Update relocate.template to include storage and container pathes in the template o Correct directory structure below the /db2/<SID>/sapdataX filesystems Update db2nodes.cfg (in the directory of the source instance)
When the script is installed, a .conf file needs to be created in the directory /usr/sap/hostctrl/exe/operations.d so that the Host Agent can start the script.
cat ISICC_PreSAPinst.conf Name: ISICC_PreSAPinst Command: /usr/sap/hostctrl/exe/operations.d/IBM/PreSAPinst.ksh $[SYSTEM_HOST:#required] $[system_sid:#required] $[SERVICE_CLASS:#required] $[service_name:#required] $[service_type:#required] Workdir: /usr/sap/hostctrl/exe/operations.d/IBM/traces ResultConverter: flat Platform: Unix
In the .conf file, the script and its parameters are defined in the Command: section. Workdir: contains the actual work directory for the script. Make sure that the directory exists. Restart the SAP Host Agent to register the script.
5.2.8.2 Define the Provider Implementation Definition
For the Provider Implementation Definition, login to SAP LVM and navigate to SetupExtensibilityCustom Operations, Hooks, and NotificationsProvider Implementation Definition. Click the Add button to start a new definition.
Page 143 of 156
Figure 116 Provider Implementation Definition (I) On the following screen: 1. Enter name of the hook and select Type Script Registered with Host Agent
2. The hostname running a SAP Host Agent where the script was registered 3. Click Retrieve Registered Scripts
Figure 117 Provider Implementation Definition (II)
Page 144 of 156
Now, select the registered script from the pick list.
Figure 118 Provider Implementation Definition (III) The registered script is a custom hook, and it shall be started for an instance (Database, SAP Central Instance). Checkmark Instance in the row for Hook. Afterwards, save the new definition.
Figure 119 Provider Implementation Definition (IV)

5.2.8.3 Define the Custom Hook Definition
The Custom Hook Definition then maps the script defined before to the Pre-SAPInst Event. Change to the Custom Hook Definition Tab (SetupExtensibilityCustom Operations, Hooks, and NotificationsCustom Hooks), and click Add.
Figure 120 Custom Hook Definition (I)
Page 145 of 156
On the next panel: 1. Enter a name for the custom hook 2. Select the provider implementation definition defined before from the Definition Name list. 3. Select Instance from the Entity Type list. 4. Select the operation SAPinst from the Operation pick list.
Figure 121 Custom Hook Definition (II) 5. Select the hook type Pre Hook from the list. The hook is called directly as pre-event before starting the SAPInst rename step. Add an OS Type (static) constraint in the table at the bottom to ensure the hook runs on AIX hosts only. Click Save to confirm the changes.
Figure 122 Custom Hook Definition (III)
5.2.9 Destroy a SAP System During the SAP system destroy action the storage configuration and all the actual volumes for the systems will be deleted in the storage system IP labels, which were created in the DNS server during SAP system creation by SAP LVM, will be removed from the DNS server once again
The SAP system will be deleted in the SAP LVM repository.
All instances for the system needs to be in unprepared state before the Destroy action can be started in SAP LVM. If the instances are still running, then the system needs to be stopped, and an unprepare operation needs to be started for it first.
5.2.9.1 Stop and Unprepare a SAP System
Navigate to the Operations view and check the status of the system. As a prerequisite to start the Destroy operation, all the instances of the system need to be in state Unprepared. In the example below the system is still active. A mass operation to Stop and Unprepare is selected for both the database and the SAP Central Instance.
Figure 123 System State in Operations View Click Execute to start the mass operation. The instances get locked for other operations, and the Stop and Unprepare operation is started. The progress of the current action can be monitored in the Monitoring view. On the upper section of the screen a status and progress overview is given. By selecting an operation from the list, and choosing the Steps sheet the individual steps of the workflow can be monitored.
Page 147 of 156
Figure 124 Monitoring View Wait until the Stop and Unprepare operation completes for all the instances.
Figure 125 Mass Stop and Unprepare Completed
5.2.9.2
Destroy the SAP System
Navigate to the Provisioning tab and select the system to be destroyed.
Page 148 of 156
Figure 126 Destroy System Click Destroy System: three additional selection/information steps are processed before the final destroy operation is started. During the destroy operation all the storage volumes which were deployed for the system during an install/copy/clone process are removed on the storage system. In the first step, all these storage volumes to be deleted later are shown.
Figure 127 System Destroy: Delete Storage Volumes Click Next to proceed. If SAP LVM is configured for DNS updates and has generated DNS entries for the system during a SAP LVM deployment process then these entries will get removed during the destroy operation. This feature was not configured in the test environment. Instead all IP addresses and corresponding DNS entries were preconfigured in the DNS before. So in the example case this list is empty.
Page 149 of 156
Figure 128 System Destroy: Host Names Click Next to proceed to the summary screen. The parameters are shown for a final review. Click Start System Destroy to start the actual destroy operation.
Figure 129 System Destroy: Summary
Page 150 of 156
6 Troubleshooting hints and tricks

6.1 Manual cleanup of SAP LVM copy process
Manual cleanup may be required in case one of the SAP System clone, copy, or refresh steps fails and the SAP LVM task Destroy partially prepared system does not work. SAP Note 1925060 describes the required cleanup steps. Below we list the actions that should be performed in case of cleanup. Check the above SAP note 1925060 for updates. 6.1.1 Retrieve information details about the process On SAP source host Identify the profile directory of the copy process. The directory is located within FCM Directory as configured in SAP LVM Storage Manager configuration and begins with "SAPLVM_<TARGET_HOST>_<SOURCE_HOST>_<STORAGE_ID>_". On SAP source host Go to the directory found in the previous step and check if "backup.log" is available. o Check for lines with the following pattern: "#SAVED <PREFIX>__<BACKUP_ID> <DIRECTORY>" (e.g. "#SAVED SAP0__A0HCJ22U1M /usr/sap/L43") Note the Backup ID. o In case the last line of backup.log shows "FMM0024I Return code is: 2.", the backup id is not relevant and the last step of this cleanup procedure can be skipped. o In case backup.log is not available, the last step of this cleanup procedure can be skipped. In SAP LVM UIMonitoringLogs Open SAP LVM logs of PreProcessCloneVolumes step and note target volume names from context. In SAP LVM UIConfiguration: o Go to SAP LVM mount configuration for each of the SAP target system instances. o Note mount points that should become available on SAP target host after successful process. o Check if the values for export path of the mount points are not empty. If the export path contain comma separated IDs, note them.
6.1.2 Cleanup hosts Stop SAP instances and all instance agents Stop the SAP Host Agent Clean up FCM process on SAP copy central management host o Get corresponding "acsd" process and kill it. "ps -ef | grep acsd" will show a process that uses profile directory identified in first step. The directory name will end with "_acsd". (e.g. "kill -9 12345") o Delete or move profile directory identified in first step. (e.g. "mv /home/sapadm/acs/SAPLVM_* /usr/sap/hostctrl/exe/operations.d/IBM/traces") Clean up FCM process on SAP copy source host o Get corresponding "acsgen" process and kill it. "ps -ef | grep acsgen" will show a process that uses profile directory identified in first step.
Delete or move profile directory identified in first step. (e.g. "mv /home/sapadm/acs/SAPLVM_* /usr/sap/hostctrl/exe/operations.d/IBM/traces") Clean up FCM process on SAP copy target host o Get corresponding "acsgen" process and kill it. "ps -ef | grep acsgen" will show a process that uses profile directory identified in first step. o Delete or move profile directory we found previously. (e.g. "mv /home/sapadm/acs/SAPLVM_* /usr/sap/hostctrl/exe/operations.d/IBM/traces" Clean up firewall rules on SAP copy target host o Check if SAP LVM related firewall rules are available. SAP LVM related firewall rules begin with "SLM" in rule description. "lsfilt -v4 -O" will show all currently active firewall rules. Delete all SAP LVM related firewall rules Delete the "deny" rules first, otherwise the SAP target host may become isolated. "rmfilt -v4 -n xxx" will delete the firewall rule with ID xxx. Activate updated firewall rule list afterwards Use "mkfilt -v4 -u" to activate the changes. Clean up storage file systems on SAP copy target host o Check with "df -g" if any of file systems retrieved previously from SAP LVM mount configuration of SAP target system instances are mounted. o Unmount all of these file systems (e.g. "umount -f /usr/sap/CPY"). o Get available logical volume group names with "lsvg". o Identify with "lsvg -l <VGNAME>" all the logical volume groups that include file systems retrieved previously from SAP LVM mount configuration of SAP target system instances. o Execute "varyoffvg <VGNAME>" and "exportvg <VGNAME>". Prerequisite: all the file systems listed in logical volume groups with "lsvg -l <VGNAME>" are closed. o Execute "lspv" and get all the physical disks on the system. o Remove all the physical disks that do not belong to any logical volume groups from the operating system by executing "rmdev -d -l hdiskX" Clean-up /usr/sap/sapservices Remove all entries set by SAP LVM on the target host o /etc/hosts o /etc/services Restart SAP Host Agent
6.1.3 Cleanup storage Identify storage volumes and/or snapshots that should be removed In order to identify storage volumes / snapshots use target volume names and/or Backup ID and/or IDs from export path in SAP LVM mount configuration retrieved previously. Target volume names correspond to storage volume / snapshot names, Backup ID will be a part of snapshot name (on XIV) and IDs from export path in SAP LVM mount configuration correspond to storage volume / snapshot WWNs (on XIV, V7000 or on SVC) Ensure that the storage volumes / snapshots are NOT masked This process is very specific and depends on the storage subsystem or storage cluster. Below we describe only actions that must be performed but not the exact process itself.
Page 152 of 156
6.2
Check if storage volumes / snapshots are attached to host alias / port IDs that represent SAP target host. o Ensure that you completed clean up storage file systems procedure on SAP copy target host. Otherwise, you will need to restart SAP target host because of AIX volume group errors. o Detach storage volumes / snapshots from host alias / port IDs that represent SAP target host. On V7000 storage system or on SVC cluster: remove consistency groups and FlashCopy relations that include storage volumes to be deleted: o Remove FlashCopy relations from consistency group o Remove consistency group o Bring up FlashCopy relations into a final state (e.g. stop them if "Stop" action is available) o Delete FlashCopy relation. Set optional flag "Delete the FlashCopy mapping even when the data on the target volume is inconsistent, or if the target volume has other dependencies." Delete storage volumes and/or snapshots. Delete the System in SAP LVM
Go to SAP LVM UIConfigurationSystems Select the appropriate system, and press Remove Selected Instances and Systems (Figure 130). Confirm the deletion in the following conformation dialog.
Figure 130 Remove System in SAP LVM
Page 153 of 156
7 Resources
7.1 IBM Documentation
7.1.1 Whitepapers Flyer Cloud-enabled SAP application management IBM Cloud Solution for SAP Integrating IBM Infrastructure with SAP Landscape Virtualization Management Introduction to an Integral IBM / SAP Cloud Solution for SAP Landscape Management IBM entry cloud configuration for SAP solutions on Power Basic Concepts and High Level Design 7.1.2 Redbooks / Technotes Deploying Cloud Components on POWER IBM Flex System Manager IBM PureFlex System and IBM Flex System Products and Technology IBM Flex System p260 and p460 Compute Node 7.1.3 Product Documentation Requirements and support for AIX using Network Installation Manager (NIM) Requirements and support for AIX, IBM i, and Linux using storage copy services (SCS) 7.1.4 iRAM IBM Systems Director VMControl Implementation Perform Guide 7.2 SAP Documentation and SAP Notes
7.2.1 SAP Documentation SAP NetWeaver Landscape Virtualization Management SAP Community Network (SCN) Virtualization and Cloud SAP Help Portal (SAP LVM) SAP NetWeaver Landscape Virtualization Management, Enterprise SAP NetWeaver Landscape Virtualization Management at a Glance 7.2.2 SAP Notes
Page 154 of 156
IBM SAP International Competence Center SAP Note Note 962955 Note 1292836 Note 1438774 Note 1465491 Note 1527538 Note 1572841 Note 1589145 Note 1589175 Note 1614266 Note 1619720 Note 1644520 Note 1728222 Note 1728293 Title / Description Use of virtual TCP/IP host names Logfiles of the SAP Hostagent New profile parameter system/uuid and system/description Provider Implementation Definition SAP NetWeaver Landscape Virtualization Management 1.0 Setup of name server update of SAP LVM Task Manager for Technical Configuration System Copy: Task Content for Task Manager System Copy: Post Copy Automation (PCA) / LVM 1.0 System Rename for SAP Systems based on SAP NetWeaver Integration of IBM Components into SAP LVM Configuration of IBM Systems Director VMControl for SAP LVM Configuration of IBM Power HMC for SAP LVM
Table 19 List of SAP Notes
Page 155 of 156
COPYRIGHT LICENSE
If this information contains sample application programs in source language, which illustrate programming techniques on various operating platforms, you may copy, modify, and distribute these sample programs in any form without payment to IBM, for the purposes of developing, using, marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore, cannot guarantee or imply reliability, serviceability, or function of these programs. The information contained in this document has not been submitted to any formal IBM test and is provided "AS IS" with no warranties or guarantees either expressed or implied. All examples cited or described in this document are presented as illustrations of the manner in which some IBM products can be used and the results that may be achieved. Actual environmental costs and performance characteristics will vary depending on individual client configurations and conditions. IBM is not responsible for printing errors in this document that result in pricing or information inaccuracies. A full list of U.S. trademarks owned by IBM may be found at: http://www.ibm.com/legal/copytrade.shtml A full list of trademarks owned by SAP may be found at: http://www.sap.com/company/legal/copyright/trademark.epx SAP and other SAP products and services mentioned herein, as well as their respective logos, are trademarks or registered trademarks of SAP AG in Germany and in several other countries all over the world. UNIX is a registered trademark in the United States, other countries or both. Linux is a registered trademark of Linus Torvalds in the United States, other countries or both. Microsoft, Windows, Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries or both.
Page 156 of 156

Integration of IBM Infrastructure and SAP LVM v2.0

Загружено:

Сведения о документе

Исходное описание:

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

Integration of IBM Infrastructure and SAP LVM v2.0

Загружено:

Авторское право:

Доступные форматы

IBM SAP International Competence Center

IBM SAP Technical Brief

IBM SAP International Competence Center Walldorf, Germany

Version: 2.0 Status: December 2013

IBM SAP International Competence Center

IBM SAP International Competence Center

IBM SAP International Competence Center

IBM SAP International Competence Center

IBM SAP International Competence Center

IBM SAP International Competence Center

IBM SAP International Competence Center

IBM SAP International Competence Center

IBM SAP International Competence Center

IBM SAP International Competence Center

IBM SAP International Competence Center

IBM SAP International Competence Center

2 Overview and Planning

IBM SAP International Competence Center

IBM SAP International Competence Center

Figure 4 IBM Systems Director and Plug-Ins

IBM SAP International Competence Center

IBM SAP International Competence Center

Figure 5 Software structure to integrate IBM IT layers

IBM SAP International Competence Center

IBM SAP International Competence Center

IBM SAP International Competence Center

IBM SAP International Competence Center

Landscape Overview System monitoring

VM (LPAR) Activate Deactivate Relocate

VM (LPAR) Provision new LPARs

SAP LVM Adapter

IBM libraries Not required

Standard Edition SAP LVM Edition

SAP System Clone Copy Refresh

IBM storage and platform libraries

Yes, requires VMControl Express Edition

Yes, requires VMControl Express Edition

Yes, requires VMControl Enterprise Edition and Storage Control

Required IBM XIV IBM SVC IBM V7000 IBM DS8000

IBM SAP International Competence Center

3-4 days working with customer

Two weeks working with customer

Two weeks with customer

At least 2 weeks with customer

Verification Steps using the customers SAP system

At least 2 weeks working with customer

Document Solution or Handover

at least 2 days for handover, documentation at least 1 week

IBM SAP International Competence Center

are installed and configured.

IBM Flex System Manager VM Control, Storage Manager

SAP Target AIX

IBM XIV SVC / Storwize

VIOS vFC (NPIV)

Storage connected via vFC (NPIV) Compliant Storage Layout

IBM SAP International Competence Center

IBM SAP International Competence Center

IBM SAP International Competence Center

IBM SAP International Competence Center

IBM XIV Storage connected via vFC (NPIV)

FC adapter port 0 vFC Server

Figure 8 Redundant virtual fibre channel attachment

AIX operating system specifics