PowerSystems For AIX-III

V5.
cover
Front cover
Power Systems for AIX III: Advanced Administration and Problem Determination
(Course code AN15)
Student Notebook
ERC 1.1
Student Notebook
Trademarks The reader should recognize that the following terms, which appear in the content of this training document, are official trademarks of IBM or other companies: IBM is a registered trademark of International Business Machines Corporation. The following are trademarks of International Business Machines Corporation in the United States, or other countries, or both: AIX HACMP POWER4 POWER6 Power Systems Redbooks System i Tivoli AIX 5L MWAVE POWER5 POWER Gt1 PowerVM RS/6000 System p WebSphere DB2 POWER POWER5+ POWER Gt3 pSeries SP System p5 Workload Partitions Manager
Adobe is either a registered trademark or a trademark of Adobe Systems Incorporated in the United States, and/or other countries. Java and all Java-based trademarks and logos are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. Windows is a trademark of Microsoft Corporation in the United States, other countries, or both. UNIX is a registered trademark of The Open Group in the United States and other countries. Other company, product, or service names may be trademarks or service marks of others.
November 2009 edition

The information contained in this document has not been submitted to any formal IBM test and is distributed on an as is basis without any warranty either express or implied. The use of this information or the implementation of any of these techniques is a customer responsibility and depends on the customers ability to evaluate and integrate them into the customers operational environment. While each item may have been reviewed by IBM for accuracy in a specific situation, there is no guarantee that the same or similar results will result elsewhere. Customers attempting to adapt these techniques to their own environments do so at their own risk.
Copyright International Business Machines Corporation 2009. All rights reserved. This document may not be reproduced in whole or in part without the prior written permission of IBM. Note to U.S. Government Users Documentation related to restricted rights Use, duplication or disclosure is subject to restrictions set forth in GSA ADP Schedule Contract with IBM Corp.
V5.3
Student Notebook
TOC
Contents
Trademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi Course description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii Agenda . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvii Unit 1. Advanced AIX administration overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-1 Unit objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-2 Application outages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-3 Live Partition Mobility versus Live Application Mobility . . . . . . . . . . . . . . . . . . . . . . 1-5 Maintenance window tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-7 Effective problem management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-10 Before problems occur . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-12 Before problems occur: A few good commands . . . . . . . . . . . . . . . . . . . . . . . . . . 1-14 Steps in problem resolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-15 Progress and reference codes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-18 Working with AIX Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-21 AIX Support test case data (1 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-23 AIX Support test case data (2 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-25 AIX software update hierarchy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-26 Relevant documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-28 Checkpoint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-29 Exercise 1: Advanced AIX administration overview . . . . . . . . . . . . . . . . . . . . . . . 1-30 Unit summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-31 Unit 2. The Object Data Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-1 Unit objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-2 2.1. Introduction to the ODM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-3 What is the ODM? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-4 Data managed by the ODM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-5 ODM components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-7 ODM database files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-8 Device configuration summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-10 Configuration manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-11 Location and contents of ODM repositories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-12 How ODM classes act together . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-14 Data not managed by the ODM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-15 Lets review: Device configuration and the ODM . . . . . . . . . . . . . . . . . . . . . . . . . 2-16 ODM commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-17 Changing attribute values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-19 Using odmchange to change attribute values . . . . . . . . . . . 2-21 2.2. ODM database files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-23 Software vital product data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-24 Software states you should know about . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-26
Copyright IBM Corp. 2009

Course materials may not be reproduced in whole or in part without the prior written permission of IBM.
Contents
iii
Student Notebook
Predefined devices (PdDv) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-28 Predefined attributes (PdAt) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-32 Customized devices (CuDv) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-34 Customized attributes (CuAt) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-37 Additional device object classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-38 Checkpoint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-40 Exercise 3: The Object Data Manager (ODM) . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-41 Unit summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-42 Unit 3. Error monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-1 Unit objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-2 3.1. Working with the error log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-3 Error logging components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-4 Generating an error report using SMIT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-6 The errpt command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-9 A summary report (errpt) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-11 A detailed error report (errpt -a) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-12 Types of disk errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-14 LVM error log entries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-16 Maintaining the error log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-17 Exercise 2: Error monitoring (part 1) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-19 3.2. Error notification and syslogd. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-21 Error notification methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-22 Self-made error notification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-24 ODM-based error notification: errnotify . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-26 syslogd daemon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-29 syslogd configuration examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-31 Redirecting syslog messages to error log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-34 Directing error log messages to syslogd . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-35 System hang detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-36 Configuring shdaemon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-38 Exercise 2: Error monitoring (part 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-40 3.3. Resource monitoring and control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-41 Resource monitoring and control (RMC) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-42 RMC conditions property screen: General tab . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-44 RMC conditions property screen: Monitored Resources tab . . . . . . . . . . . . . . . . .3-45 RMC actions property screen: General tab . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-46 RMC actions property screen: When in Effect tab . . . . . . . . . . . . . . . . . . . . . . . . .3-47 RMC management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-48 Exercise 2: Error monitoring (part 3) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-50 Checkpoint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-51 Unit summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-52 Unit 4. Network Installation Manager basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4-1 Unit objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4-2 NIM overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4-3 Machine roles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4-5 Boot process for AIX installation (tape or CD) . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4-7
iv AIX Advanced Administration
V5.3
Student Notebook
TOC
Boot process for AIX installation (network) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-9 NIM objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-11 Listing NIM objects and their attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-13 NIM configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-14 resources objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-16 resources objects: lpp_source . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-18 resources objects: spot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-21 resources objects: mksysb . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-24 networks objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-26 machines objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-28 Defining a machine object . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-30 Define a client using SMIT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-32 NIM operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-34 bos_inst operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-38 More information about NIM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-40 Additional topics in NIM course . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-45 Exercise 4 overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-46 Checkpoint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-47 Unit summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-48 Unit 5. System initialization: Part I. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-1 Unit objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-3 5.1. System startup process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-5 How does a System p server or LPAR boot? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-6 Loading of a boot image . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-8 Contents of the boot logical volume (hd5) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-10 5.2. Unable to find boot image. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-13 Working with bootlists . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-14 Starting System Management Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-16 Working with bootlists in SMS (1 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-18 Working with bootlists in SMS (2 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-20 5.3. Corrupted boot logical volume . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-21 Boot device alternatives (1 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-22 Boot device alternatives (2 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-24 Accessing a system that will not boot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-25 Booting in maintenance mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-28 Working in maintenance mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-29 How to fix a corrupted BLV . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-31 Checkpoint (1 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-33 Checkpoint (2 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-34 Exercise 3: System initialization: Part 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-35 Unit summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-36 Unit 6. System initialization: Part II . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Unit objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.1. AIX initialization part 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . System software initialization overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . rc.boot 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6-1 6-2 6-3 6-4 6-6

v
Contents
Student Notebook
rc.boot 2 (part 1) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6-8 rc.boot 2 (part 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6-10 rc.boot 3 (part 1) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6-12 rc.boot 3 (part 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6-14 rc.boot summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6-16 Fixing corrupted file systems and logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6-17 Lets review: rc.boot (1 of 3) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6-19 Lets review: rc.boot (2 of 3) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6-20 Lets review: rc.boot (3 of 3) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6-21 6.2. AIX initialization part 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6-23 Configuration manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6-24 Config_Rules object class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6-26 cfgmgr output in the boot log using alog . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6-28 /etc/inittab file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6-29 Boot problem management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6-31 Lets review: /etc/inittab file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6-34 Checkpoint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6-36 Exercise 4: System initialization part 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6-37 Unit summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6-38 Unit 7. Disk management theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-1 Unit objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-2 7.1. LVM data representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-3 LVM terms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-4 LVM identifiers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-6 LVM data on disk control blocks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-8 LVM data in the operating system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-10 Contents of the VGDA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-11 VGDA example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-13 The logical volume control block (LVCB) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-16 How LVM interacts with ODM and VGDA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-18 ODM entries for physical volumes (1 of 3) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-20 ODM entries for physical volumes (2 of 3) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-22 ODM entries for physical volumes (3 of 3) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-23 ODM entries for volume groups (1 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-24 ODM entries for volume groups (2 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-25 ODM entries for logical volumes (1 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-26 ODM entries for logical volumes (2 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-27 ODM-related LVM problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-28 Fixing ODM problems (1 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-30 Fixing ODM problems (2 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-32 Intermediate level ODM commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-35 Exercise 7: LVM metadata and problems (parts 1 and 2) . . . . . . . . . . . . . . . . . . .7-37 7.2. Failed disks: Mirroring and quorum issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-39 Mirroring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-40 Stale partitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-42 Mirroring rootvg . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-44 VGDA count . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-46
vi AIX Advanced Administration
V5.3
Student Notebook
TOC
Quorum not available . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Nonquorum volume groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Forced vary on (varyonvg -f) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Physical volume states . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Checkpoint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Exercise 7: LVM Metadata and problems (parts 4 and 5) . . . . . . . . . . . . . . . . . . . Unit summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7-47 7-49 7-51 7-53 7-55 7-56 7-57
Unit 8. Disk management procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-1 Unit objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-3 8.1. Disk replacement techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-5 Disk replacement: Starting point . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-6 Procedure 1: Disk mirrored . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-8 Procedure 2: Disk still working . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-10 Procedure 2: Special steps for rootvg . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-12 Procedure 3: Disk in missing or removed state . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-14 Procedure 4: Total rootvg failure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-16 Procedure 5: Total non-rootvg failure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-18 Frequent disk replacement errors (1 of 4) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-20 Frequent disk replacement errors (2 of 4) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-21 Frequent disk replacement errors (3 of 4) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-22 Frequent disk replacement errors (4 of 4) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-23 8.2. Export and import . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-25 Exporting a volume group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-26 Importing a volume group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-28 importvg and existing logical volumes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-30 importvg and existing file systems (1 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-31 importvg and existing file systems (2 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-33 Checkpoint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-35 Exercise 8: Exporting and importing volume groups . . . . . . . . . . . . . . . . . . . . . . . 8-36 Unit summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-37 Unit 9. Install and backup techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9-1 Unit objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9-2 9.1. Alternate disk installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9-3 Topic 1 objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9-4 Alternate disk installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9-5 Alternate mksysb disk installation (1 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9-8 Alternate mksysb disk installation (2 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9-10 Alternate disk rootvg cloning (1 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9-11 Alternate disk rootvg cloning (2 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9-12 Removing an alternate disk installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9-13 NIM alternate disk migration (nimadm) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9-15 Exercise 9, topic 1: Alternate disk install . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9-17 9.2. Using multibos . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9-19 Topic 2 objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9-20 multibos overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9-21 Active and standby BOS logical volumes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9-23
Contents
vii
Student Notebook
Setting up a standby BOS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9-24 Other multibos operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9-26 Exercise 9, topic 2: multibos . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9-29 9.3. JFS2 snapshot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9-31 Topic 3 objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9-32 JFS2 snapshot (1 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9-33 JFS2 snapshot (2 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9-35 JFS2 snapshot mechanism (1 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9-37 JFS2 snapshot mechanism (2 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9-38 JFS2 snapshot SMIT menu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9-39 Creating snapshots (external) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9-40 Creating snapshots (internal) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9-43 Listing snapshots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9-44 Using a JFS2 snapshot to recover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9-45 Using a JFS2 snapshot to back up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9-47 JFS2 snapshot space management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9-48 Exercise 9, topic 3: JFS2 snapshot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9-49 Checkpoint (1 of 4) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9-50 Checkpoint (2 of 4) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9-51 Checkpoint (3 of 4) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9-52 Checkpoint (4 of 4) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9-53 Unit summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9-54 Unit 10. Workload partitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-1 Unit objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-2 10.1. Workload partitions review. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-3 Topic 1 objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-4 AIX workload partitions (WPAR) review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-5 System WPAR and application WPAR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-8 System WPAR file systems space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-10 10.2. WPAR Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-13 Topic 2 objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-14 Workload Partition Manager overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-15 Workload Partition Manager main GUI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-17 WPAR Manager topology: Default configuration . . . . . . . . . . . . . . . . . . . . . . . . .10-19 Installation and configuration: WPAR Manager . . . . . . . . . . . . . . . . . . . . . . . . . .10-21 Installation and configuration: WPAR agent . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-24 Authentication and WPAR Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-26 WPAR Manager functional view . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-28 Basic management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-30 Creating a WPAR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-31 WPAR monitoring and reporting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-32 Resources view . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-33 Manual relocation or mobility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-34 Tasks activity and logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-35 WPAR 1.2 log locations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-37 10.3. Application mobility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-39 Topic 3 objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-40
viii AIX Advanced Administration
V5.3
Student Notebook
TOC
Application mobility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . WPAR Manager relocation support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Compatibility issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Live partition mobility versus live application mobility . . . . . . . . . . . . . . . . . . . . . WPAR enhanced live mobility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Steps for WPAR enhanced live mobility (WPAR Mgr GUI) . . . . . . . . . . . . . . . . . Enhanced relocation workflow (1 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Enhanced relocation workflow (2 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Enhanced relocation error (1 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Enhanced relocation error (2 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Steps for WPAR enhanced live mobility (command line) . . . . . . . . . . . . . . . . . . Enhanced live relocation: CLI (1 of 4) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Enhanced live relocation: CLI (2 of 4) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Enhanced live relocation: CLI (3 of 4) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Enhanced live relocation: CLI (4 of 4) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Steps for WPAR static relocation (WPAR Mgr GUI) . . . . . . . . . . . . . . . . . . . . . . Steps for checkpoint and restart relocation: CLI . . . . . . . . . . . . . . . . . . . . . . . . . Checkpoint and restart relocation: CLI (1 of 3) . . . . . . . . . . . . . . . . . . . . . . . . . . Checkpoint and restart relocation: CLI (2 of 3) . . . . . . . . . . . . . . . . . . . . . . . . . . Checkpoint and restart relocation: CLI (3 of 3) . . . . . . . . . . . . . . . . . . . . . . . . . . Checkpoint (1 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Checkpoint (2 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Unit summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
10-41 10-42 10-44 10-46 10-48 10-50 10-52 10-53 10-54 10-55 10-56 10-57 10-58 10-59 10-62 10-63 10-65 10-67 10-68 10-69 10-71 10-72 10-73
Unit 11. The AIX system dump facility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-1 Unit objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-2 System dumps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-3 Types of dumps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-4 How a system dump is invoked . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-6 LED 888 code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-8 When a dump occurs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-10 The sysdumpdev command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-11 Dedicated dump device (1 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-16 Dedicated dump device (2 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-17 Estimating dump size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-19 dumpcheck utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-21 Methods of starting a dump . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-23 Start a dump from a TTY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-26 Generating dumps with SMIT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-28 Dump-related LED codes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-29 Copying system dump . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-31 Automatically reboot after a crash . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-33 Sending a dump to IBM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-35 Use kdb to analyze a dump . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-38 Checkpoint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-41 Exercise 11: System dump . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-42 Unit summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-43

Contents
ix
Student Notebook
Appendix A. Checkpoint solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-1 Appendix B. Command summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B-1 Appendix C. AIX dump code and progress codes. . . . . . . . . . . . . . . . . . . . . . . . . . . C-1 Appendix D. Auditing security related events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D-1 Appendix E. Diagnostics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . E-1
AIX Advanced Administration

V5.3
Student Notebook
TMK
Trademarks
The reader should recognize that the following terms, which appear in the content of this training document, are official trademarks of IBM or other companies: IBM is a registered trademark of International Business Machines Corporation. The following are trademarks of International Business Machines Corporation in the United States, or other countries, or both: AIX HACMP POWER4 POWER6 Power Systems Redbooks System i Tivoli AIX 5L MWAVE POWER5 POWER Gt1 PowerVM RS/6000 System p WebSphere DB2 POWER POWER5+ POWER Gt3 pSeries SP System p5 Workload Partitions Manager
Adobe is either a registered trademark or a trademark of Adobe Systems Incorporated in the United States, and/or other countries. Java and all Java-based trademarks and logos are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. Windows is a trademark of Microsoft Corporation in the United States, other countries, or both. UNIX is a registered trademark of The Open Group in the United States and other countries. Other company, product, or service names may be trademarks or service marks of others.

Trademarks
xi
Student Notebook
xii

V5.3
Student Notebook
pref
Course description
Power Systems for AIX III: Advanced Administration and Problem Determination Duration: 5 days Purpose
This course provides advanced AIX system administrator skills with a focus on availability and problem determination. It provides detailed knowledge of the ODM database where AIX maintains so much configuration information. It shows how to monitor for and deal with AIX problems. There is special focus on dealing with Logical Volume Manager problems, including procedures for replacing disks. Several techniques for minimizing the system maintenance window are covered. It also covers how to migrate AIX Workload Partitions to another system with minimal disruption. While the course includes some AIX 6.1 enhancements, most of the material is applicable to prior releases of AIX.
Audience
This is an advanced course for AIX system administrators, system support, and contract support individuals with at least six months of experience in AIX.
Prerequisites
You should have basic AIX System Administration skills. These skills include: Use of the Hardware Management Console (HMC) to activate a logical partition running AIX and to access the AIX system console Install an AIX operating system from an already configured NIM server Implementation of AIX backup and recovery Manage additional software and base operating system updates Familiarity with management tools such as SMIT Understand how to manage file systems, logical volumes, and volume groups

Course description
xiii
Student Notebook
Understand basic Workload Partition (WPAR) concepts and commands (recommended for the WPAR Manager content) Mastery of the UNIX user interface including use of the vi editor, command execution, input and output redirection, and the use of utilities such as grep These skills could be developed through experience or by formal training. Recommended training courses to obtain these prerequisite skills are either of the following: Power Systems for AIX III: Advanced Administration and Problem Determination (AN12) and its prerequisites AIX System Administration I: Implementation (AU14) and its prerequisites. (Note that AU14 does not cover WPARs) If the student has AIX system administration skills, but is not familiar with the LPAR environment, those skills may be obtained by attending either of the following: AU73/Q1373 System p Virtualization I: Planning and Configuration AN11 Power Systems Administration I: LPAR Configuration
Objectives
On completion of this course, students should be able to: Perform system problem determination and reporting procedures including analyzing error logs, creating dumps of the system, and providing needed data to the AIX Support personnel Examine and manipulate Object Data Manager databases Identify and resolve conflicts between the Logical Volume Manager (LVM) disk structures and the Object Data Manager (ODM) Complete a very basic configuration of Network Installation Manager to provide network boot support for either system installation or booting to maintenance mode Identify various types of boot and disk failures and perform the matching recovery procedures Implement advanced methods such as alternate disk install, multibos, and JFS2 snapshots to use a smaller maintenance window Install and configure Workload Partition Manager to support WPAR management and to implement Live Application Mobility (LAM)
xiv

V5.3
Student Notebook
pref
Contents
Overview of advanced administration techniques Error monitoring The Object Data Manager (ODM) Basic Network Installation Manager (NIM) configuration System initialization problem determination Disk management theory and procedures Advanced techniques for installation and backup Workload Partition (WPAR) Manager and Live Application Mobility The AIX system dump facility

Course description
xv
Student Notebook
xvi

V5.3
Student Notebook
pref
Agenda
Day 1
Welcome Unit 1 - Advanced AIX administration overview Exercise 1 - Problem diagnostic information Unit 2 - The Object Data Manager Exercise 2 - The Object Data Manager Unit 3 - Error monitoring Exercise 3 - Error monitoring
Day 2
Unit 4 - Network Installation Manager basics Exercise 4 - Basic NIM configuration Unit 5 - System initialization: Part I Exercise 5 - System initialization: Part I (optional) Exercise 3 Part 3 - Using RMC to monitor resources on a system
Day 3
Unit 6 - System initialization: Part II Exercise 6 - System initialization: Part: II Unit 7 - Disk management theory Exercise 7 - LVM metadata and problems Unit 8 - Disk management procedures Exercise 8 parts 1 and 2: Disk replacement techniques (optional) Exercise 7 part 5 - Manually fixing an LVM ODM problem
Day 4
Unit 8, Part 2 - Export and import (to fix VGDA/ODM conflict) Exercise 8 parts 3 and 4 - Disk management procedures Unit 9 - Install and backup techniques Exercise 9, part 1 - Alternate disk copy (pre-clone) Unit 9, topic 2 - multibos Exercise 9, part 1 - Wait for clone completion (30 min clone) Exercise 9, part 1 - Alternate disk copy (post-clone) Exercise 9, part 2 - multibos (pre-clone) Unit 9, topic 3 - JFS2 snapshot Exercise 9, part 2: wait for clone completion (37 min clone Exercise 9, part 2: multibos (post-clone)
Agenda
xvii
Student Notebook
Exercise 9, part 3: JFS2 snapshot Unit 10, topic 1 - Workload partitions review Unit 10, topic 2 - WPAR Manager Exercise 10 part 1 - Installing WPAR Manager (optional) Exercise 7 part 3 - Using intermediate LVM commands
Day 5
Exercise 10 part 2 - Create and activate a WPAR Unit 10, topic 3 - Application mobility Exercise 10 part 3 - Enhanced Live Application Mobility Exercise 10 part 4- Working with static relocation Unit 11 - The AIX system dump facility Exercise 11 - System dump facility (optional) Exercise 10 part 4 - Working with static relocation Wrap up / Evaluations
xviii AIX Advanced Administration

V5.3
Student Notebook
Uempty
Unit 1. Advanced AIX administration overview

What this unit is about
This unit introduces various AIX administration issues related to problem determination and handling system maintenance and backup in an efficient manner.
What you should be able to do

After completing this unit you should be able to: List the steps of a basic methodology for problem determination List AIX features that assist in minimizing planned downtime or shortening the maintenance window Explain how to find documentation and other key resources needed for problem resolution
How you will check your progress

Accountability: Checkpoint questions Lab exercise
References
SG24-5496 SG24-5766 SG24-7559 Problem Solving and Troubleshooting in AIX 5L (Redbook) AIX 5L Differences Guide Version 5.3 Edition (Redbook) IBM AIX Version 6.1 Differences Guide (Redbook)

1-1
Student Notebook
Unit objectives
IBM Power Systems
After completing this unit, you should be able to: List the steps of a basic methodology for problem determination List AIX features that assist in minimizing planned downtime or shortening the maintenance window Explain how to find documentation and other key resources needed for problem resolution
Copyright IBM Corporation 2009
Figure 1-1. Unit objectives
AN151.0
Notes:
1-2

V5.3
Student Notebook
Uempty
Application outages
IBM Power Systems
Functional or performance Avoid unplanned outages with best practices

Change control Data security Capacity planning High availability design
Avoid planned outages

Fall-over to backup server Relocate application (LPAR or WPAR mobility)
Use maintenance windows

Application stopped versus slow activity Plan enough time for back-out or recovery Minimize time needed
Effective problem determination and recovery

Figure 1-2. Application outages
AN151.0
Notes: Introduction
Providing system availability is a major responsibility of any system administrator. An outage may be caused by a functional problem (such as an application or system crash) or a server performance problem (business is seriously impacted due to poor response times or late jobs). There are many approaches to dealing with this.
Unplanned outages
When most of us think of availability, we think of unplanned outages. Regular hardware and software maintenance can often avoid these outages. Designing the computing facility to have redundant components (power, network adapters, network switches, storage, and more) can make the overall system resilient to the failure of individual components. Performance problems are often the result of failing to do proper capacity planning, resulting in not enough resources (memory, processors, network bandwidth, or disk I/O bandwidth) to handle the increased workload. If there is no change control to manage what
Copyright IBM Corp. 2009 Unit 1. Advanced AIX administration overview
1-3
Student Notebook
work is placed on a system, capacity planning is even more challenging. Furthermore, uncontrolled changes to a system result in uncontrolled exposure to possible outages created by those changes, an thus unplanned outages. Computer viruses and other malicious attacks by computer hackers can also reduce system availability (in addition to the exposure of losing proprietary information). Good data security policies are essential. Even when implementing good policies in these areas, some unplanned outages will still happen. In these situations, the system administrator needs to have a plan for minimizing the impact and recovering as quickly as possible. One common approach is to have an alternate system that can take over the work of the failed system. High Availability Cluster Multi-Processing (HACMP) provides a system for either concurrent processing by multiple systems, or an automated fall-over to a backup system, thus minimizing the impact of a server failure. Such server redundancy can be designed to work within a single facility or be divided between different geographical locations. Obviously, rapid notification of a problem, effective and prompt diagnosis of the cause, and being able to quickly implement an effective solution will all contribute to a smaller mean time to recovery.
Planned outages
By using change control, the risk associated with certain categories of potential unplanned outages can be managed by implementing the changes during planned windows of time when the impact of any unexpected problem (resulting from the change) is minimized. In addition, there are certain types of changes for which an outage is unavoidable. Some facilities will implement multiple types of maintenance windows. One type would be frequent short maintenance windows for any administrative work that will compete with applications for resources (performance impact) or have a small chance of having a functional disruption. Another type would be a less frequent window in which any reboot of the system or any major change to the level of the operating system or major subsystems, such as database software, would be allowed. Sometimes, the amount of time in a maintenance window is relatively small and the work has to be carefully planned. You also need to allow time to recover if any thing goes wrong due to the maintenance. Any needed resources that can be pre-staged will help expedite the work. Any approach that can speed recovery after a problem occurs is also useful. For systems which need to be up 24 hours a day, seven days a week, and every day in the year (24x7x365), even a short outage cannot be tolerated. In those situation, a method to non-disruptively move the applications to another system can be invaluable. If an HACMP cluster solution is already in place to handle unplanned outages, then this can be used to manually fall-over the services to another system while maintenance is being done. Other solutions are to use Live Partition Mobility or Live Application Mobility.
1-4

V5.3
Student Notebook
Uempty
Live Partition Mobility versus Live Application Mobility

IBM Power Systems
Live Partition Mobility allows the Multiple systems managed by a single HMC migration of a running logical partition to another physical server. Server 1 Server 2 Operating system, applications, P1 P2 P3 P1 P5 and services are not stopped during the process Requires POWER6 , AIX 5.3 HMC and VIO server Network
VIOS
Live Application Mobility allows moving a workload partition from one server to another. Without requiring the workload running in the AIX # 2 WPAR to be restarted Provides outage avoidance Workload 2. AIX # 1 1. Partition and multi-system Workload Billing AIX # 3 Partition workload balancing Workload Workload Workload Data Mining n Partition Partitio Partition Test EMail App Srv ad rklo Requires AIX 6.1 Wo tition
Workload Partition Web Par Training Workload Partition Dev
Policy Workload Partitions Manager
Figure 1-3. Live Partition Mobility versus Live Application Mobility
VIOS
AN151.0
Notes:
As the number of hosted partitions and applications increases, finding a maintenance window acceptable to all becomes increasingly difficult. Live partition or application mobility allow you to move your partitions around such that you can perform disruptive operations on the machine when it best suits you, rather than when it causes the least inconvenience to the users.
Live Partition Mobility

Live Partition Mobility provides the ability to move a running logical partition (including its operating system and applications) non-disruptively from one system to another. The migration operation, which takes just a few seconds, maintains complete system transactional integrity. The migration transfers the entire system environment, including processor state, memory, attached virtual devices, and connected users.
Live Application Mobility

1-5
Student Notebook
Live Application Mobility (LAM) is a new capability that allows a client to relocate a running WPAR from one system to another, without requiring the workload running in the WPAR to be restarted. LAM is intended for use within a data center and requires the use of the new Licensed Program Product, the IBM AIX Workload Partitions Manager. Live Application Mobility differs significantly from Live Partition Mobility in that Live Partition Mobility is a feature of POWER6 processors. As such, it can be used on operating systems other than AIX 6, such as Linux or earlier AIX versions. On the other hand, WPAR is specifically a feature of AIX 6, but it can run on various hardware platforms (for example: POWER6, POWER5 or POWER5+, or POWER4 systems).
1-6

V5.3
Student Notebook
Uempty
Maintenance window tasks

IBM Power Systems
Minimize time needed for tasks Operating system maintenance

Pre-staging of maintenance Applying maintenance to alternate rootvg Applying maintenance with alternate BLV Reboot to use updated alternate
System backups
Minimizing rootvg size Snapshot techniques for user file systems
Figure 1-4. Maintenance window tasks
AN151.0
Notes: Expediting work in the maintenance window

The quicker maintenance can be completed the sooner you can get the system back up and head home (this is likely at night or on a weekend). More importantly, expediting the expedited activities will allow more time to handle any problems that may arise.
Operating system maintenance

Ensure you have, on hand, whatever materials you will need for the job, such as the installation media. Eliminating the need to handle that media can be important. This can be done by pre-copying all of the needed filesets to disk storage. This could be on an NFS or NIM server (provided you have sufficient network bandwidth) or it could be a software repository on the system being updated. If using a software repository on the system which is being updated, it is recommended that the filesets be in a file system allocated out of a different volume group than the rootvg.

1-7
Student Notebook
An important technique, that we will cover, is the use of an alternate storage for the target of the software update. What we mean is that the updates are not made to the rootvg, but rather to a copy of the rootvg. This has two advantages. First, there is no change being made to the active rootvg. For locations that make a distinction between changing the level of the operating system and simply doing work that has a performance impact, the actual time consuming update activity can be done in a more frequently available window. Then when a major maintenance window arrives, you only need to reboot to make it effective. The second advantage, and to some the more important advantage, is the ease of recovery. If you find that there are serious problems with running under the new level of code, you only need to reboot back to the earlier code level, rather than recover from a mksysb or reject the entire update. Of course, the down side is that you will need to reboot to make the update effective; but, this is something a major maintenance window should expect. There are two techniques that we will cover. One technique, is creating an alternate set of logical volumes that are copies of the rootvg BOS logical volumes. This is called multibos. The other technique, is creating an alternate volume group which is a clone of the rootvg. In each case, you would apply the maintenance to the copy and then later reboot to make it effective.
Expediting backups
Another common maintenance activity is backing up the system. Unless you have an application that is designed to manage a recovery process using fuzzy backups, you will need to quiesce the application activity long enough to be sure that there are no inconsistencies in the backup. The term fuzzy backup refers to a backup in which the application was making changes during the backup. For a given transaction, multiple data changes are made. Some of these transaction related changes are made before that data was backed up, while other changes were made after that data was backed up. Thus the backup has one piece of data which reflects the transaction and another piece of data that does not reflect the transaction. The two pieces of data are inconsistent and such a backup is referred to as fuzzy. For the rootvg itself, the size of the rootvg should be minimized. It should only contain what is needed for the OS. All user data and other non-essential files should be backed up and restored separately. An example would be the standard location of a software repository: /usr/sys/inst.images. The software repository can be very large and yet this common path resides in the /usr file system, which is in the rootvg. Placing the software repository in a separate file system with its own recovery plan (could be using the original media as the backup) can help reduce backup and recovery time. Another common example is the /home filesystem. If users have vast amounts of data stored there, then over mounting with a separate file system can again speed up working with the rootvg. There other file systems such as /tmp that could have contents be eliminated from the system backup.The trick is that these would need to be excluded (not mounted or identified in /etc/exclude.rootvg) from the backup during mksysb execution, and then
1-8

V5.3
Student Notebook
Uempty
separately recovered from their own backup. Other user data will be in separate user volume groups. With the emphasis on separate backups for non-BOS data, there comes a need to minimize how long the applications need to be quiesced and still have data consistency. One technique that AIX provides is JFS2 snapshots, which will allow us to only very briefly quiesce the application and still have a consistent picture of the data at a single point in time. Then we can either use that snapshot of the data as its own backup, or base an actual backup upon that snapshot (in order to have off-site storage of the backup). There other facilities for doing snapshot captures of data. Some are part of the storage subsystems and some are part of total storage solutions such as Tivoli Storage Manager. Our focus will be on the facility that is provided with AIX: JSF2 snapshot.

1-9
Student Notebook
Effective problem management

IBM Power Systems
Keep system documentation current Keep maintenance up to date. Use a problem determination methodology. If an AIX bug:
Collect problem information. Open problem report with AIX Support. Provide snap with information.
Figure 1-5. Effective problem management
AN151.0
Notes: Obtaining and documenting information about your system

It is a good idea, whenever you approach a new system, to learn as much as you can about that system. It is also critical to document not only the physical resources and the devices, but also how the system has been configured (network, LVM, and more). Then this information will be ready when needed. Later in the course, we will suggest some ways to collect system information.
System maintenance
Sometimes code works well under normal testing or production circumstances, but can have a poor logic discovered when faced with an unanticipated situation. Alternatively, it could be some non-central aspect of the code that is not noticed normally. The number of facilities using this code is large enough that there is a good chance that one of the facilities will detect and report the problem not long after release of the new code level.
1-10 AIX Advanced Administration
V5.3
Student Notebook
Uempty
The fix for the code defect will usually come out in the next released fix pack. On the other hand, many facilities may not be effected by or be concerned about the code defect problem for months, until the circumstances arise in which it represents a problem. By installing newer service packs, a facility can benefit from the experience of others and avoid being impacted by known problems. Obviously there is always the possible exposure that a new fix pack will introduce new problems, while solving many old problems. This course will cover some techniques to use in applying fix packs.
Problem determination
Once you find yourself impacted by what you believe to be a product defect, you will need to obtain prompt resolution. While there is no substitute for experience (the ability to recognize a situation and remember the details of how you dealt with it the last time a similar problem occurred), many problems will be most effectively solved by following a well developed problem determination methodology. This course will cover a basic problem determination methodology.
Problem determination
When you find yourself impacted by what you believe to be a product defect, you will need to contact AIX Support. Before contacting AIX Support, you should write up a description of the problem and the surrounding circumstances. When you open a new Problem Management Report (PMR) with AIX Support, you will be expected to provide them with a wealth of information to assist them in determining the cause of the problem. The snap command is a common tool to assist in collecting a vast amount of information about the environment surrounding the problem. The course materials will cover these problem reporting procedures.

1-11
Student Notebook
Before problems occur

IBM Power Systems
Effective problem determination starts with a good understanding of the system and its components. The more information you have about the normal operation of a system, the better.
System configuration Operating system level Applications installed Baseline performance Installation, configuration, and service manuals
System System documentation documentation
Figure 1-6. Before problems occur
AN151.0
Notes: Obtaining and documenting information about your system

It is a good idea, whenever you approach a new system, to learn as much as you can about that system. It is also critical to document both logical and physical device information so that it is available when troubleshooting is necessary.
Information that should be documented

Examples of important items that should be determined and recorded include the following: - Machine architecture (model, CPU type) - Physical volumes (type and size of disks)

V5.3
Student Notebook
Uempty
- Volume groups (names, just a bunch of disks (JBOD) or redundant array of independent disks (RAID) - Logical volumes (mirrored or not, which VG, type) - Filesystems (which VG, what applications) - Memory (size) and paging spaces (how many, location)

1-13
Student Notebook
Before problems occur: A few good commands

IBM Power Systems
lspv lscfg prtconf lsvg lsps lsfs lsdev getconf bootinfo snap
Lists physical volumes, PVID, VG membership Provides information regarding system components Displays system configuration information Lists the volume groups Displays information about paging spaces Gives file system information Provides device information Displays values of system configuration variables Displays system configuration information (unsupported) Collects system data
Figure 1-7. Before problems occur: A few good commands
AN151.0
Notes: A list of useful commands

The list of commands on the visual provides a starting point for use in gathering key information about your system. There are also many other commands that can help you in gathering important system information.
Sources of additional information

Be sure to check the man pages or the AIX Commands Reference for correct syntax and option flags to be used with these commands to provide more specific information. There is no man page or entry in the AIX Commands Reference for the bootinfo command.

V5.3
Student Notebook
Uempty
Steps in problem resolution

IBM Power Systems
1.Identify the problem 2. Talk to users to define the problem 3. Collect system data 4. Resolve the problem
Figure 1-8. Steps in problem resolution
AN151.0
Notes: The start-to-finish method

The start-to-finish method for resolving problems consists primarily of the following four major components: Identify the problem. Talk to users (to define the problem). Collect system data. Resolve (fix) the problem.
Step 1: Identify the problem

The first step in problem resolution is to find out what the problem is. It is important to understand exactly what the users of the system perceive the problem to be. A clear description of the problem typically gives clues as to the cause of the problem and aids in the choice of troubleshooting methods to apply.
Copyright IBM Corp. 2009 Unit 1. Advanced AIX administration overview
1-15
Student Notebook
Step 2: Gathering additional detail

A problem might be identified by just about anyone who has use of or a need to interact with the system. If a problem is reported to you, it may be necessary to get details from the reporting user and then query others on the system in order to obtain additional details or to develop a clear picture of what happened. The users may be data entry staff, programmers, system administrators, technical support personnel, management, application developers, operations staff, network users, and so forth.
Suggested questions
What is the problem? What is the system doing (or not doing)? How did you first notice the problem? When did it happen? Have any changes been made recently?
Keep them talking until the picture is clear. Ask as many questions as you need to in order to get the entire history of the problem.
Step 3 - Collect system data

Some information about the system will have already been collected from the users during the process of defining the problem. By using various commands, such as lsdev, lspv, lsvg, lslpp, lsattr, and others, you can gather further information about the system configuration. You should also gather other relevant information by making use of available error reporting facilities, determining the state of the operating system, checking for the existence of a system dump, and inspecting the various available log files. How is the machine configured? What errors are being produced? What is the state of the OS? Is there a system dump? What log files exist?
SMIT and Web-based system manager logs

If SMIT and the Web-based System Manager have been used, there will be additional logs that could provide further information. These log files are normally contained in the home directory of the root user and are named (by default) /smit.log for SMIT and /websm.log for the Web-based System Manager.

V5.3
Student Notebook
Uempty
Step 4 - Resolve the problem

After all the information is gathered, determine the procedures necessary to solve the problem. Keep a log of all actions you perform in trying to determine the cause of the problem, and any actions you perform to correct the problem. - Use the information gathered. - Keep a log of actions taken to correct the problem. - Use the tools available: commands documentation, downloadable fixes, and updates. - Contact IBM Support, if necessary.
Resources for problem solving

A variety of resources, such as the documentation for individual commands, are available to assist you in solving problems with AIX 6 systems. The IBM System p and AIX Information Center is a Web site that serves as a focal point for all information pertaining to pSeries and AIX. It provides a link to the entire pSeries library. A message database is available to search on error numbers, error identifiers, and display codes (LED values). The Web site also contains FAQs, how-tos, a Troubleshooting Guide, and more.
Information Center URL

The URL for the IBM System p and AIX Information Center is as follows: http://publib16.boulder.ibm.com/pseries/index.htm

1-17
Student Notebook
Progress and reference codes

IBM Power Systems
Progress codes System reference codes (SRCs) Service request numbers (SRNs) Obtained from:
Front panel of system enclosure HMC or IVM (for logically partitioned systems) Operator console message or diagnostics (diag utility)
Online hardware and AIX documentation available at: http://publib.boulder.ibm.com/infocenter/systems

Select System Hardware > System i and System p
Popular links and effective searches available
Select Operating System > AIX 6.1 Information

Search for message center Diagnostic Information for Multiple Bus Systems (SA38-0509)
Figure 1-9. Progress and reference codes
AN151.0
Notes: Introduction
AIX provides progress and error indicators (display codes) during the boot process. These display codes can be very useful in resolving startup problems. Depending on the hardware platform, the codes are displayed on the console and the operator panel.
Operator panel
For non-LPAR systems, the operator panel is an LED display on the front panel. POWER4, POWER5, and POWER6-based systems can be divided into multiple Logical Partitions (LPARs). In this case, a system-wide LED display still exists on the front panel. However, the operator panel for each LPAR is displayed on the screen of the Hardware Management Console (HMC). The HMC is a separate system which is required when running multiple LPARs. Regardless of where they are displayed, they are often referred to as LED Display Codes.

V5.3
Student Notebook
Uempty
Progress codes and other reference codes

Reference codes can have various sources: - Diagnostics: Diagnostics or error log analysis can provide Service Request Numbers (SRNs) which can be used to determine the source of a hardware or operating system problem. - Hardware initialization: System firmware sends boot status codes (called firmware checkpoints) to the operator panel. Once the console is initialized, the firmware can also send 8-digit error codes to the console. - AIX initialization: The rc.boot script and the device configuration methods send progress and error codes to the operator panel. Codes from the hardware/firmware or from AIX initialization scripts fall into two categories: - Progress Codes: These are checkpoints indicating the stages in the initial program load (IPL) or boot sequence. They do not necessarily indicate a problem unless the sequence permanently stops on a single code or a rotating sequence of codes. - System Reference Codes (SRC): These are error codes indicating that a problem has originated in hardware, Licensed Internal Code (firmware), or in the operating system.
Documentation
Note: all information on Web sites and their design is based upon what is available at the time of this course revision. Web site URLs and the design of the related Web pages often change. Online hardware documentation and AIX message codes are available at: http://publib.boulder.ibm.com/infocenter/systems - Many of the codes you will deal with are actually hardware or firmware related. For those codes, you need to navigate to the infocenter that specializes in system hardware. The content area has popular links for accessing code information, or you can use search strings such as: system reference codes, service request numbers, or service support troubleshooting. - For AIX codes and messages, you will need to navigate to the Operating System infocenter for AIX.

1-19
Student Notebook
From here you can use the search string of AIX message center to obtain information on various codes (including the seven digit message codes). One very useful reference that you can find at the AIX infocenter is the: RS/6000 Eserver pSeries Diagnostic Information for Multiple Bus Systems (SA38-0509). Chapter 30 has AIX diagnostic numbers and location codes. It provides descriptions for the numbers and characters that display on the operator panel and descriptions of the location codes used to identify a particular item.

V5.3
Student Notebook
Uempty
Working with AIX Support

IBM Power Systems
Have needed information ready:

Name, phone #, customer #, Machine type model and serial #, AIX version, release, technology level, and service pack Problem description, including error codes Severity level: critical, significant impact, some impact, minimal
1-800-IBM-SERV (1-800-426-7378) Level 1 will collect information and assign PMR number Route to level 2 responsible for the product You may be asked to collect additional information to upload They may ask you to update to a specific TL or SP
APAR for your problem already addressed Need to have a standard environment for them to investigate
Figure 1-10. Working with AIX Support
AN151.0
Notes:
If you believe that your problem is the result of a system defect, you can call AIX Support to request assistance. Before you call 1-800-IBM-SERV, it is a good idea to have certain information ready. They will want to verify your name against a list of names associated with your customer number, and validate that your customer number has support for the product in question. They will also need to know some details about the hardware and software environment in which the problem is occurring - such as your MTMS (machine type, model, serial), your AIX OS level, and the level of any other relevant software. Of course, you need to explain your problem, providing as much detail as possible, especially any error messages or codes. The level 1 personnel will ask you for the priority of your problem. Severity level 1(critical) indicates that the function does not work, your business is severely impacted, there is no work around, and that there needs to be an immediate solution. Be aware that, for severity level 1, you will be expected to be available 24x7 until the problem is resolved.

1-21
Student Notebook
Severity level 2 (significant impact) indicates that the function is usable but is limited in a way that your business is severely impacted. Severity level 3 (some impact) indicates that the program is usable with less significant features (not critical to operations) unavailable. Severity level 4 (minimal impact) indicates that the problem causes little impact on operations, or a reasonable circumvention to the problem has been implemented. Level 1 will assign you a PMR number (actually a PMR and branch number combination) for tracking purposes. Each time, in the future, when you call about this problem, you should have the PMR and branch numbers at hand. Once the basic information has been collected, you are passed to level 2 personal for the product area for which you are having a problem. They will work with you in investigating the nature and cause of your problem. They will search the support database to see if it is a known problem that is either already being worked on or has a solution already developed. In many cases, they will request that you update to a specific technology level and service pack that already includes the fix. If they do not have a fix, they may still ask you to update your system and determine if the problem still exists. If the problem still exists, they now have a known software environment to work with. At this point they will often ask for a complete set of information from your system to be collected and uploaded to their server, to support their investigation. The basic tool for collecting your system information is the snap command.

V5.3
Student Notebook
Uempty
AIX Support test case data (1 of 2)

IBM Power Systems
Run the following (or very similar) commands to gather snap information:
# snap a <Copy any extra data to the /tmp/ibmsupt/testcase or the /tmp/ibmsupt/other directory.> # snap c
This step will create /tmp/ibmsupt/snap.pax.Z.
# mv /tmp/ibmsupt/snap.pax.Z \ PMR#.b<branch#>.c<country#>.snap.pax.Z
Figure 1-11. AIX Support test case data (1 of 2)
AN151.0
Notes: Overview of the snap command

The snap command is used to gather system configuration information useful in identifying and resolving system problems. The snap command can also be used to compress the snap information gathered into a pax file. The file may then be written to a device such as tape or DVD, or transmitted to a remote system. Refer to the man page for snap or the corresponding entry in the AIX Commands Reference manual for detailed information about the snap command and its various flags.

1-23
Student Notebook
Discussion of command sequence shown on the visual

First, as illustrated on the visual, the -a flag of the snap command should be used to gather all system configuration information that can be gathered using snap. The output of this command will be written to the /tmp/ibmsupt directory. Next, you should place any additional testcase data that you feel may be helpful in resolving the problem being investigated into the /tmp/ibmsupt/ other subdirectory or into the /tmp/ibmsupt/testcase subdirectory. This additional information is then included (together with the information gathered directly by snap) in the compressed pax file created in the next step in this command sequence. As shown, the -c flag of the snap command should then be used to create a compressed pax file containing all files contained in the /tmp/ibmsupt directory. The output file created by this command is /tmp/ibmsupt/snap.pax.Z. Next, the /tmp/ibmsupt/snap.pax.Z output file should be renamed using the mv command to indicate the PMR number, branch number, and country number associated with the data in the file. For example, if the PMR number is 12345, the branch number is 567, and the country number is 890, the file should be renamed 12345.b567.c890.snap.pax.Z. (The country code for the United States is: 000).

V5.3
Student Notebook
Uempty
AIX Support test case data (2 of 2)

IBM Power Systems
Upload the information you have captured:

# ftp testcase.software.ibm.com User: anonymous Password: <your email address> ftp> cd /aix/toibm ftp> bin ftp> put PMR#.b<branch#>.c<country#>.snap.pax.Z ftp> quit
Figure 1-12. AIX Support test case data (2 of 2)
AN151.0
Notes: Uploading data to AIX Support

AIX Support provides an anonymous FTP server for receiving your testcase data. The host name for that server is: testcase.software.ibm.com. Once you login to the server, change directory to /aix/toibm. Be sure to transfer the file as binary to avoid an undesirable attempt by FTP to convert the contents of the file. Then just put your file on the server and notify your support contact that the data is there.

1-25
Student Notebook
AIX software update hierarchy

IBM Power Systems
Version and release (oslevel)

Requires new license and migration install
Fileset updates (lslpp L will show mod and fix levels)

Collected changes to files in a fileset Related to APARs and PTFs Only need to apply the new fileset
Fix bundles
Collections of fileset updates
Technology level and maintenance level (oslevel r)

Fix bundle of enhancements and fixes
Service packs (oslevel s)

Fix bundle of important fixes
Interim fixes
Special situation code replacements Delay for normal PTF packaging is too slow Managed with efix tool
Figure 1-13. AIX software update hierarchy
AN151.0
Notes: Version, release, mod, and fix

The oslevel command by default shows us the version and release of the operating system. Changing this requires a new license and a disruption to the system (such as rebooting to installation and maintenance to do a migration install). The mod and fix levels in the oslevel -s output are normally displayed as zeros. The mod level displayed in the oslevel output should reflect the technology level. The mod and fix levels are used to reflect changes to the many individual filesets which make up the operating system. These are best seen by browsing through the output of the lslpp -L report. These changes only require the administrator to install a Program Temporary Fix (PTF) in the form of a fix fileset. A given fix fileset can resolve one or more problems or APARs (Authorized Program Analysis Report).

V5.3
Student Notebook
Uempty
Fix bundles
It is useful to collect many accumulated PTFs together and test them together. This can then be used as a base line for a new cycle of enhancements and corrections. By testing them together, it is often possible to catch unexpected interactions between them. There are two types of AIX fix bundles. One type of fix bundle is a Technology Level (TL) update (formally known as Maintenance Level or ML). This is a major fix bundle which not only includes many fixes for code problems, but also includes minor functional enhancements. You can identify the current AIX technology level by running the oslevel -r command. Another type of bundling is a Service Pack (SP). A Service Pack is released more frequently than a Technology Level (between TL releases) and usually only contains needed fixes. You can identify the current AIX technology level and service pack by running the oslevel -s command. For the oslevel command to reflect a new TL or SP, all related filesets fixes must be installed. If a single fileset update in the fix bundle is not installed, the TL or SP level will not change.
Interim fixes
On rare occasions, a customer has an urgent situation which needs fixes for a problem so quickly that they cannot wait for the formal PTF to be released. In those situations, a developer may place one or more individual file replacements on an FTP server and allow the system administrator to download and install them. Originally, this would simply involve manually copying the new files over the old files. But this created problems, especially in identifying the state of a system which later experienced other (possibly related) problems or in backing out the changes. Today, there is a better methodology for managing these interim fixes using the efix command. Security alerts will often provide interim fixes for the identified security exposure. Depending upon your own risk analysis, you might immediately use the interim fix, or wait for the next service pack (which will include these security fixes). The syntax and use of the efix command was covered in the prerequisite course.

1-27
Student Notebook
Relevant documentation
IBM Power Systems
IBM System p and AIX Information Center entry page: http://publib.boulder.ibm.com/eserver

Links to:
IBM Systems Information Center IBM Systems Hardware Information Center IBM Systems Software Information Center IBM System p and AIX information Center
The System p and AIX information Center and links for both:
AIX 5L Version 5.3 AIX Version 6.1
IBM Redbooks home: http://www.redbooks.ibm.com

Figure 1-14. Relevant documentation
AN151.0
Notes: IBM System p and AIX Information Center

Most software and hardware documentation for AIX 5L and AIX 6 systems can be accessed online using the IBM System p and AIX Information Center Web site: http://publib16.boulder.ibm.com/pseries/index.htm
IBM systems Information Center

Hardware documentation for POWER5 processor-based systems can be accessed online using the IBM Systems Information Centers site.
IBM Redbooks
Redbooks can be viewed, downloaded, or ordered from the IBM Redbooks Web site: http://www.redbooks.ibm.com

V5.3
Student Notebook
Uempty
Checkpoint
IBM Power Systems
1. What are the four major problem determination steps? _________________________________________ _________________________________________ _________________________________________ _________________________________________ 2. Who should provide information about system problems? _________________________________________ _________________________________________ 3. True or False: If there is a problem with the software, it is necessary to get the next release of the product to resolve the problem. 4. True or False: Documentation can be viewed or downloaded from the IBM Web site.
Figure 1-15. Checkpoint
AN151.0
Notes:

1-29
Student Notebook
Exercise 1: Advanced AIX administration overview

IBM Power Systems
Recording system information Finding reference code documentation Creating a snap file
Figure 1-16. Exercise 1: Advanced AIX administration overview
AN151.0
Notes:

V5.3
Student Notebook
Uempty
Unit summary
IBM Power Systems
Having completed this unit, you should be able to: List the steps of a basic methodology for problem determination List AIX features that assist in minimizing planned downtime or shortening the maintenance window Explain how to find documentation and other key resources needed for problem resolution
Figure 1-17. Unit summary
AN151.0
Notes:

1-31
Student Notebook

V5.3
Student Notebook
Uempty
Unit 2. The Object Data Manager

This unit describes the structure of the Object Data Manager (ODM). It shows the use of the ODM command line interface and explains the role of the ODM in device configuration. Specific information regarding the function and content of the most important ODM files is also presented.

After completing this unit, you should be able to: Describe the structure of the ODM Use the ODM command line interface Explain the role of the ODM in device configuration Describe the function of the most important ODM files

Accountability: Checkpoint questions Lab exercise
References
Online Online Online AIX Version 6.1 Command Reference volumes 1-6 AIX Version 6.1 General Programming Concepts: Writing and Debugging Programs AIX Version 6.1 Technical Reference: Kernel and Subsystems
Note: References listed as online above are available through the IBM Systems Information Center at the following address: http://publib.boulder.ibm.com/infocenter/systems

2-1
Student Notebook
Unit objectives
IBM Power Systems
After completing this unit, you should be able to: Describe the structure of the ODM Use the ODM command line interface Explain the role of the ODM in device configuration Describe the function of the most important ODM files
AN151.0
Notes: Importance of this unit

The ODM is a very important component of AIX and is one major feature that distinguishes AIX from other UNIX systems. This unit describes the structure of the ODM and explains how you can work with ODM files using the ODM command line interface. It is also very important that you, as an AIX system administrator, understand the role of the ODM during device configuration. Thus, explaining the role of the ODM in this process is another major objective of this unit.
2-2

V5.3
Student Notebook
Uempty
2.1. Introduction to the ODM

2-3
Student Notebook
What is the ODM?

IBM Power Systems
The Object Data Manager (ODM) is a database intended for storing system information. Physical and logical device information is stored and maintained through the use of objects with associated characteristics.
Figure 2-2. What is the ODM?
AN151.0
Notes:
2-4

V5.3
Student Notebook
Uempty
Data managed by the ODM

IBM Power Systems
Devices
Software
System resource controller
ODM
SMIT menus
TCP/IP configuration
Error Log, Dump
NIM
Figure 2-3. Data managed by the ODM
AN151.0
Notes: System data managed by ODM

The ODM manages the following system data: - Device configuration data - Software Vital Product Data (SWVPD) - System Resource Controller (SRC) data - TCP/IP configuration data - Error log and dump information - NIM (Network Installation Manager) information - SMIT menus and commands

2-5
Student Notebook
Emphasis in this unit

Our main emphasis in this unit is on the use of ODM to store and manage information regarding devices and software products (software vital product data). During the course, many other ODM classes are described.
2-6

V5.3
Student Notebook
Uempty
ODM components
IBM Power Systems
uniquetype
tape/scsi/scsd
attribute
block_size
deflt
none
values
0-2147483648,1
disk/scsi/osdisk
pvid
none
tty/rs232/tty
login
disable
enable, disable, ...
Figure 2-4. ODM components
AN151.0
Notes: Completing the drawing on the visual

The drawing on the visual above identifies the basic components of ODM, but some terms have been intentionally omitted from the drawing. Your instructor will complete this drawing during the lecture. Please complete your own copy of the drawing by writing in the terms supplied by your instructor.
ODM data format

For security reasons, the ODM data is stored in binary format. To work with ODM files, you must use the ODM command line interface. It is not possible to update ODM files with an editor.

2-7
Student Notebook
ODM database files

IBM Power Systems
Predefined device information Customized device information Software vital product data SMIT menus Error log, alog, and dump information System resource controller Network Installation Manager (NIM)
Figure 2-5. ODM database files
PdDv, PdAt, PdCn CuDv, CuAt, CuDep, CuDvDr, CuVPD, Config_Rules history, inventory, lpp, product sm_menu_opt, sm_name_hdr, sm_cmd_hdr, sm_cmd_opt SWservAt SRCsubsys, SRCsubsvr, ... nim_attr, nim_object, nim_pdattr
AN151.0
Notes: Major ODM files

The table on the visual summarizes the major ODM files in AIX. As you can see, the files listed in this table are placed into several different categories.
Current focus
In this unit, we will concentrate on ODM classes that are used to store device information and software product data. At this point, we will narrow our focus even further and confine our discussion to ODM classes that store device information.
2-8

V5.3
Student Notebook
Uempty
Predefined and customized device information

The first two rows in the table on the visual indicate that some ODM classes contain predefined device information and that others contain customized device information. What is the difference between these two types of information? Predefined device information describes all supported devices. Customized device information describes all devices that are actually attached to the system. It is very important that you understand the difference between these two information classifications. The classes themselves are described in more detail in the next topic of this unit.

2-9
Student Notebook
Device configuration summary

IBM Power Systems
Predefined databases
PdCn
PdDv PdAt
Configuration Manager (cfgmgr)
Config_Rules
Customized databases
CuDep CuDv CuAt
CuDvDr
CuVPD
Figure 2-6. Device configuration summary
AN151.0
Notes: ODM classes used during device configuration

The visual above shows the ODM object classes used during the configuration of a device.
Roles of cfgmgr and Config_Rules

When an AIX system boots, the Configuration Manager (cfgmgr) is responsible for configuring devices. There is one ODM object class which the cfgmgr uses to determine the correct sequence when configuring devices: Config_Rules. This ODM object class also contains information about various methods files used for device management.

V5.3
Student Notebook
Uempty
Configuration manager
IBM Power Systems
Predefined
PdDv PdAt PdCn
"Plug and Play"
Config_Rules
cfgmgr
Customized
CuDv CuAt CuDep CuDvDr CuVPD
Methods
Define
Device Driver
Load
Configure Change
Unload
Unconfigure Undefine
AN151.0
Figure 2-7. Configuration manager
Notes: Importance of Config_Rules object class

Although cfgmgr gets credit for managing devices (adding, deleting, changing, and so forth), it is actually the Config_Rules object class that does the work through various methods files.

2-11
Student Notebook
Location and contents of ODM repositories

IBM Power Systems
CuDv CuAt CuDep CuDvDr CuVPD Config_Rules history inventory lpp product nim_* SWservAt SRC*
Network
PdDv PdAt PdCn history inventory lpp product sm_*
history inventory lpp product
/etc/objrepos
/usr/lib/objrepos
/usr/share/lib/objrepos
Figure 2-8. Location and contents of ODM repositories
AN151.0
Notes: Introduction
To support diskless, dataless and other workstations, the ODM object classes are held in three repositories. Each of these repositories is described in the material that follows.
/etc/objrepos
This repository contains the customized devices object classes and the four object classes used by the Software Vital Product Database (SWVPD) for the / (root) part of the installable software product. The root part of the software contains files that must be installed on the target system. To access information in the other directories, this directory contains symbolic links to the predefined devices object classes. The links are needed because the ODMDIR variable points to only /etc/objrepos. It contains the part of the product that cannot be shared among machines. Each client must have its own copy. Most of this software requiring a separate copy for each machine is associated with the configuration of the machine or product.
V5.3
Student Notebook
Uempty
/usr/lib/objrepos
This repository contains the predefined devices object classes, SMIT menu object classes, and the four object classes used by the SWVPD for the /usr part of the installable software product. The object classes in this repository can be shared across the network by /usr clients, dataless and diskless workstations. Software installed in the /usr part can be can be shared among several machines with compatible hardware architectures.
/usr/share/lib/objrepos
Contains the four object classes used by the SWVPD for the /usr/share part of the installable software product. The /usr/share part of a software product contains files that are not hardware dependent. They can be shared among several machines, even if the machines have a different hardware architecture. An example of this are terminfo files that describe terminal capabilities. As terminfo is used on many UNIX systems, terminfo files are part of the /usr/share part of a system product.
lslpp options
The lslpp command can list the software recorded in the ODM. When run with the -l (lower case L) flag, it lists each of the locations (/, /usr/lib, /usr/share/lib) where it finds the fileset recorded. This can be distracting if you are not concerned with these distinctions. Alternately, you can run lslpp -L which only reports each fileset once, without making distinctions between the root, usr, and share portions.

2-13
Student Notebook
How ODM classes act together

IBM Power Systems
PdDv: type = "14106902" class = "adapter" subclass = "pci" prefix = "ent" cfgmgr DvDr = "pci/goentdd" Define = /usr/lib/methods/define_rspc" Configure = "/usr/lib/methods/cfggoent" uniquetype = "adapter/pci/14106902"
CuDv: name = "ent1" status = 1 chgstatus = 2 ddins = "pci/goentdd" location = "02-08" parent = "pci2" connwhere = "8 PdDvLn = "adapter/pci/14106902"
PdAt: uniquetype = "adapter/pci/14106902" attribute = "jumbo_frames" deflt = "no" values = "yes,no"
chdev -l ent1 \ -a jumbo_frames=yes
CuAt: name = "ent1" attribute = "jumbo_frames" value = "yes" type = "R"
Figure 2-9. How ODM classes act together
AN151.0
Notes: Interaction of ODM classes

The visual above and the notes below summarize how ODM classes act together. 1. In order for a particular device to be defined in AIX, the device type must be predefined in ODM class PdDv. 2. A device can be defined by either the cfgmgr (if the device is detectable), or by the mkdev command. Both commands use the define method to generate an instance in ODM class CuDv. The configure method is used to load a specific device driver and to generate an entry in the /dev directory. Notice the link PdDvLn from CuDv back to PdDv. 3. At this point you only have default attribute values in PdAt which, in our example of a gigabit Ethernet adapter, means you could not use jumbo frames (default is no). If you change the attributes, for example, jumbo_frames to yes, you get an object describing the nondefault value in CuAt.
V5.3
Student Notebook
Uempty
Data not managed by the ODM

IBM Power Systems
Filesystem information
? ? ?
User/security information
Queues and queue devices
Figure 2-10. Data not managed by the ODM
AN151.0
Notes: Completion of this page

The visual above identifies some types of system information that are not managed by the ODM, but the names of the files that store these types of information have been intentionally omitted from the visual. Your instructor will complete this visual during the lecture. Please complete your own copy of the visual by writing in the file names supplied by your instructor.

2-15
Student Notebook
Lets review: Device configuration and the ODM

IBM Power Systems
_______
1.
Undefined
Defined
Available
2.
3.
AIX kernel Applications
D____ D____ 4.
/____/_____ 5.
Figure 2-11. Lets review: Device configuration and the ODM
AN151.0
Notes: Instructions
Please answer the following questions by writing them on the picture above. If you are unsure about a question, leave it out. 1. Which command configures devices in an AIX system? Note: This is not an ODM command.)Which ODM class contains all devices that your system supports? 2. Which ODM class contains all devices that are configured in your system? 3. Which programs are loaded into the AIX kernel to control access to the devices? 4. If you have a configured tape drive rmt1, which special file do applications access to work with this device?

V5.3
Student Notebook
Uempty
ODM commands
IBM Power Systems
Object class: odmcreate, odmdrop Descriptors: odmshow
uniquetype
tape/scsi/scsd
attribute
block_size
deflt
none
values
0-2147483648,1
disk/scsi/osdisk
pvid
none
tty/rs232/tty
login
disable
enable, disable, ...
Objects: odmadd, odmchange, odmdelete, odmget

Figure 2-12. ODM commands
AN151.0
Notes: Introduction
Different commands are available for working with each of the ODM components: object classes, descriptors, and objects.
Commands for working with ODM classes

1. You can create ODM classes using the odmcreate command. This command has the following syntax: odmcreate descriptor_file.cre The file descriptor_file.cre contains the class definition for the corresponding ODM class. Usually, these files have the suffix .cre. The exercise for this unit contains an optional part that shows how to create self-defined ODM classes.

2-17
Student Notebook
2. To delete an entire ODM class, use the odmdrop command. The odmdrop command has the following syntax: odmdrop -o object_class_name The name object_class_name is the name of the ODM class you want to remove. Be very careful with this command. It removes the complete class immediately.
A command for working with ODM descriptors

To view the underlying layout of an object class, use the odmshow command: odmshow object_class_name The visual shows an extraction from ODM class PdAt, where four descriptors are shown (uniquetype, attribute, deflt, and values).
Commands for working with objects

Usually, system administrators work with objects. The odmget command retrieves object information from an existing object class. To add new objects, use odmadd. To delete objects, use odmdelete. To change objects, use odmchange. Working on the object level is explained in more detail on the following pages.
The ODMDIR environment variable

All ODM commands use the ODMDIR environment variable, which is set in the file /etc/environment. The default value of ODMDIR is /etc/objrepos.

V5.3
Student Notebook
Uempty
Changing attribute values

IBM Power Systems
# odmget -q"uniquetype=tape/scsi/scsd and attribute=block_size" PdAt > file # vi file
PdAt: uniquetype = "tape/scsi/scsd" attribute = "block_size" deflt = 512" values = "0-2147483648,1" width = "" type = "R" generic = "DU" rep = "nr" nls_index = 6
Modify deflt to 512
# odmdelete -o PdAt -q"uniquetype=tape/scsi/scsd and attribute=block_size" # odmadd file

Figure 2-13. Changing attribute values
AN151.0
Notes: Discussion of command sequence on the visual

The odmget command in the example will pick all the records from the PdAt class, where uniquetype is equal to tape/scsi/scsd and attribute is equal to block_size. In this instance, only one record should be matched. The information is redirected into a file which can be changed using an editor. In this example, the default value for the attribute block_size is changed to 512. Note: Before the new value of 512 can be added into the ODM, the old object (which had the block_size set to a null value) must be deleted, otherwise you would end up with two objects describing the same attribute in the database. The first object found will be used, and the results could be quite confusing. This is why it is important to delete an entry before adding a replacement record. The final operation is to add the file into the ODM.

2-19
Student Notebook
Need to use ODM commands

The ODM objects are stored in a binary format; that means you need to work with the ODM commands to query or change any objects.
Possible queries
As with any database, you can perform queries for records matching certain criteria. The tests are on the values of the descriptors of the objects. A number of tests can be performed: = != > >= < <= like equal not equal greater greater than or equal to less than less than or equal to similar to; finds patterns in character string data
For example, to search for records where the value of the lpp_name attribute begins with bosext1., you would use the syntax lpp_name like bosext1.* Tests can be linked together using normal boolean operations, as shown in the following example: uniquetype=tape/scsi/scsd and attribute=block_size In addition to the * wildcard, a ? can be used as a wildcard character.

V5.3
Student Notebook
Uempty
Using odmchange to change attribute values

IBM Power Systems
# odmget -q"uniquetype=tape/scsi/scsd and attribute=block_size" PdAt > file # vi file
PdAt: uniquetype = "tape/scsi/scsd" attribute = "block_size" deflt = 512" values = "0-2147483648,1" width = "" type = "R" generic = "DU" rep = "nr" nls_index = 6
Modify deflt to 512
# odmchange -o PdAt -q"uniquetype=tape/scsi/scsd and attribute=block_size" file
Figure 2-14. Using odmchange to change attribute values
AN151.0
Notes: Another way of changing attribute values

The series of steps shown on this visual shows how the odmchange command can be used instead of the odmadd and odmdelete steps shown in the previous example to modify attribute values.

2-21
Student Notebook

V5.3
Student Notebook
Uempty
2.2. ODM database files

2-23
Student Notebook
Software vital product data

IBM Power Systems
lpp: name = "bos.rte.printers size = 0 state = 5 ver = 6 rel = 1 mod =0 fix = 0 description = "Front End Printer Support lpp_id = 38 inventory: lpp_id = 38 private = 0 file_type = 0 format = 1 loc0 = "/etc/qconfig loc1 = " loc2 = " size = 0 checksum = 0
Figure 2-15. Software vital product data
product: lpp_name = "bos.rte.printers comp_id = "5765-C3403 state = 5 ver = 6 rel = 1 mod =0 fix = 0 ptf = " prereq = "*coreq bos.rte 5.1.0.0 description = " supersedes = "" history: lpp_id = 38 ver = 6 rel = 1 mod = 0 fix = 0 ptf = " state = 1 time = 1187714064 comment = ""
AN151.0
Notes: Role of installp command

Whenever installing a product or update in AIX, the installp command uses the ODM to maintain the Software Vital Product Database (SWVPD).
Contents of SWVPD
The following information is part of the SWVPD: The name of the software product (for example, bos.rte.printers) The version, release, modification, and fix level of the software product (for example, 5.3.0.10 or 6.1.0.0) The fix level, which contains a summary of fixes implemented in a product Any program temporary fix (PTF) that has been installed on the system The state of the software product: - Available (state = 1)
V5.3
Student Notebook
Uempty
Applying (state = 2) Applied (state = 3) Committing (state = 4) Committed (state = 5) Rejecting (state = 6) Broken (state = 7)
SWVPD classes
The Software Vital Product Data is stored in the following ODM classes: lpp The lpp object class contains information about the installed software products, including the current software product state and description. The inventory object class contains information about the files associated with a software product. The product object class contains product information about the installation and updates of software products and their prerequisites. The history object class contains historical information about the installation and updates of software products.
inventory product
history

2-25
Student Notebook
Software states you should know about

IBM Power Systems
Applied Committed Applying, committing, rejecting, deinstalling Broken
Only possible for PTFs or Updates Previous version stored in /usr/lpp/Package_Name Rejecting update recovers to saved version Committing update deletes previous version Removing committed software is possible No return to previous version If installation was not successful: a) installp -C b) smit maintain_software
Cleanup failed Remove software and reinstall
Figure 2-16. Software states you should know about
AN151.0
Notes: Introduction
The AIX software vital product database uses software states that describe the status of an install or update package.
The applied and committed states

When installing a program temporary fix (PTF) or update package, you can install the software into an applied state. Software in an applied state contains the newly installed version (which is active) and a backup of the old version (which is inactive). This gives you the opportunity to test the new software. If it works as expected, you can commit the software, which will remove the old version. If it does not work as planned, you can reject the software, which will remove the new software and reactivate the old version. Install packages cannot be applied. These will always be committed.

V5.3
Student Notebook
Uempty
Once a product is committed, if you would like to return to the old version, you must remove the current version and reinstall the old version.
States indicating installation problems

If an installation does not complete successfully, for example, if the power fails during the install, you may find software states like applying, committing, rejecting, or deinstalling. To recover from this failure, execute the command installp -C or use the SMIT fastpath smit maintain_software. Select Clean Up After Failed or Interrupted Installation when working in SMIT.
The broken state

After a cleanup of a failed installation, you might detect a broken software status. In this case, the only way to recover from the failure is to remove and reinstall the software package.

2-27
Student Notebook
Predefined devices (PdDv)

IBM Power Systems
PdDv: type = scsd" class = "tape" subclass = "scsi" prefix = "rmt" ... base = 0 ... detectable = 1 ... led = 2418 setno = 54 msgno = 0 catalog = "devices.cat" DvDr = "tape" Define = "/etc/methods/define" Configure = "/etc/methods/cfgsctape" Change = "/etc/methods/chggen" Unconfigure = "/etc/methods/ucfgdevice" Undefine = "etc/methods/undefine" Start = "" Stop = "" ... uniquetype = "tape/scsi/scsd"
Figure 2-17. Predefined devices (PdDv)
AN151.0
Notes: The predefined devices (PdDv) object class

The Predefined Devices (PdDv) object class contains entries for all devices supported by the system. A device that is not part of this ODM class cannot be configured on an AIX system. Key attributes of objects in this class are described in the following paragraphs.
type
This specifies the product name or model number, for example, 8 mm (tape).
class
Specifies the functional class name. A functional class is a group of device instances sharing the same high-level function. For example, tape is a functional class name representing all tape devices.
V5.3
Student Notebook
Uempty
subclass
Device classes are grouped into subclasses. The subclass scsi specifies all tape devices that may be attached to a SCSI interface.
prefix
This specifies the Assigned Prefix in the customized database, which is used to derive the device instance name and /dev name. For example, rmt is the prefix name assigned to tape devices. Names of tape devices would then look like rmt0, rmt1, or rmt2.
base
This descriptor specifies whether a device is a base device or not. A base device is any device that forms part of a minimal base system. During system boot, a minimal base system is configured to permit access to the root volume group (rootvg) and hence to the root file system. This minimal base system can include, for example, the standard I/O diskette adapter and a SCSI hard drive. The device shown on the visual is not a base device. This flag is also used by the bosboot and savebase commands, which are introduced later in this course.
detectable
This specifies whether the device instance is detectable or undetectable. A device whose presence and type can be determined by the cfgmgr, once it is actually powered on and attached to the system, is said to be detectable. A value of 1 means that the device is detectable, and a value of 0 that it is not (for example, a printer or tty).
led
This indicates the value displayed on the LEDs when the configure method begins to run. The value stored is decimal, but the value shown on the LEDs is hexadecimal (2418 is 972 in hex).
setno, msgno
Each device has a specific description (for example, SCSI Tape Drive) that is shown when the device attributes are listed by the lsdev command. These two descriptors are used to look up the description in a message catalog.

2-29
Student Notebook
catalog
This identifies the filename of the national language support (NLS) catalog. The LANG variable on a system controls which catalog file is used to show a message. For example, if LANG is set to en_US, the catalog file /usr/lib/nls/msg/en_US/devices.cat is used. If LANG is de_DE, catalog /usr/lib/nls/msg/de_DE/devices.cat is used.
DvDr
This identifies the name of the device driver associated with the device (for example, tape). Usually, device drivers are stored in directory /usr/lib/drivers. Device drivers are loaded into the AIX kernel when a device is made available.
Define
This names the define method associated with the device type. This program is called when a device is brought into the defined state.
Configure
This names the configure method associated with the device type. This program is called when a device is brought into the available state.
Change
This names the change method associated with the device type. This program is called when a device attribute is changed through the chdev command.
Unconfigure
This names the unconfigure method associated with the device type. This program is called when a device is unconfigured by rmdev -l.
Undefine
This names the undefine method associated with the device type. This program is called when a device is undefined by rmdev -l -d.
Start, stop
Few devices support a stopped state (only logical devices). A stopped state means that the device driver is loaded, but no application can access the device. These two attributes name the methods to start or stop a device.

V5.3
Student Notebook
Uempty
uniquetype
This is a key that is referenced by other object classes. Objects use this descriptor as a pointer back to the device description in PdDv. The key is a concatenation of the class, subclass, and type values.

2-31
Student Notebook
Predefined attributes (PdAt)

IBM Power Systems
PdAt: uniquetype = "tape/scsi/scsd" attribute = "block_size" deflt = "" values = "0-2147483648,1" ... PdAt: uniquetype = "disk/scsi/osdisk" attribute = "pvid" deflt = "none" values = "" ... PdAt: uniquetype = "tty/rs232/tty" attribute = "term" deflt = "dumb" values = "" ...
Figure 2-18. Predefined attributes (PdAt)
AN151.0
Notes: The predefined attribute (PdAt) object class

The Predefined Attribute (PdAt) object class contains an entry for each existing attribute for each device represented in the PdDv object class. An attribute is any device-dependent information, such as interrupt levels, bus I/O address ranges, baud rates, parity settings, or block sizes. The extract out of PdAt that is given on the visual shows three attributes (block size, physical volume identifier, and terminal name) and their default values. The meanings of the key fields shown on the visual are described in the paragraphs that follow.
uniquetype
This descriptor is used as a pointer back to the device defined in the PdDv object class.

V5.3
Student Notebook
Uempty
attribute
This identifies the name of the attribute. This is the name that can be passed to the mkdev or chdev command. For example, to change the default name of dumb to ibm3151 for tty0, you can issue the following command: # chdev -l tty0 -a term=ibm3151
deflt
This identifies the default value for an attribute. Nondefault values are stored in CuAt.
values
This identifies the possible values that can be associated with the attribute name. For example, allowed values for the block_size attribute range from 0 to 2147483648, with an increment of 1.

2-33
Student Notebook
Customized devices (CuDv)

IBM Power Systems
CuDv: name = "ent1" status = 1 chgstatus = 2 ddins = "pci/goentdd" location = "02-08" parent = "pci2" connwhere = "8" PdDvLn = "adapter/pci/14106902" CuDv: name = "hdisk2" status = 1 chgstatus = 2 ddins = "scdisk" location = "01-08-01-8,0" parent = "scsi1" connwhere = "8,0" PdDvLn = "disk/scsi/scsd"
Figure 2-19. Customized devices (CuDv)
AN151.0
Notes: The customized devices (CuDv) object class

The Customized Devices (CuDv) object class contains entries for all device instances defined in the system. As the name implies, a defined device object is an object that a define method has created in the CuDv object class. A defined device object may or may not have a corresponding actual device attached to the system. The CuDv object class contains objects that provide device and connection information for each device. Each device is distinguished by a unique logical name. The customized database is updated twice, during system bootup and at run time, to define new devices, remove undefined devices, and update the information for a device that has changed. The key descriptors in CuDv are described in the next few paragraphs.

V5.3
Student Notebook
Uempty
name
A customized device object for a device instance is assigned a unique logical name to distinguish the device from other devices. The visual shows two devices, an Ethernet adapter ent1 and a disk drive hdisk2.
status
This identifies the current status of the device instance. Possible values are: - status = 0 - Defined - status = 1 - Available - status = 2 - Stopped
chgstatus
This flag tells whether the device instance has been altered since the last system boot. The diagnostics facility uses this flag to validate system configuration. The flag can take these values: - chgstatus = 0 - New device - chgstatus = 1 - Don't care - chgstatus = 2 - Same - chgstatus = 3 - Device is missing
ddins
This descriptor typically contains the same value as the Device Driver Name descriptor in the Predefined Devices (PdDv) object class. It specifies the name of the device driver that is loaded into the AIX kernel.
location
Identifies the AIX location of a device. The location code is a path from the system unit through the adapter to the device. In case of a hardware problem, the location code is used by technical support to identify a failing device.
parent
Identifies the logical name of the parent device. For example, the parent device of hdisk2 is scsi1.

2-35
Student Notebook
connwhere
Identifies the specific location on the parent device where the device is connected. For example, the device hdisk2 uses the SCSI address 8,0.
PdDvLn
Provides a link to the device instance's predefined information through the uniquetype descriptor in the PdDv object class.

V5.3
Student Notebook
Uempty
Customized attributes (CuAt)

IBM Power Systems
CuAt: name = "ent1" attribute = "jumbo_frames" value = "yes" ... CuAt: name = "hdisk2" attribute = "pvid" value = "00c35ba0816eafe50000000000000000" ...
Figure 2-20. Customized attributes (CuAt)
AN151.0
Notes: The customized attribute (CuAt) object class

The Customized Attribute (CuAt) object class contains customized device-specific attribute information. Devices represented in the Customized Devices (CuDv) object class have attributes found in the Predefined Attribute (PdAt) object class and the CuAt object class. There is an entry in the CuAt object class for attributes that take customized values. Attributes taking the default value are found in the PdAt object class. Each entry describes the current value of the attribute.
Discussion of examples on visual

The sample CuAt entries on the visual show two attributes that have customized values. The attribute login has been changed to enable. The attribute pvid shows the physical volume identifier that has been assigned to disk hdisk0.
Copyright IBM Corp. 2009 Unit 2. The Object Data Manager
2-37
Student Notebook
Additional device object classes

IBM Power Systems
PdCn: uniquetype = "adapter/pci/sym875 connkey = "scsi connwhere = "1,0" PdCn: uniquetype = "adapter/pci/sym875 connkey = "scsi connwhere = "2,0"
CuDvDr: resource value1 = value2 = value3 = CuDvDr: resource value1 = value2 = value3 =
= "devno" "36" "0" "hdisk3 = "devno" "36" "1" "hdisk2"
CuDep: name = "rootvg dependency = "hd6" CuDep: name = "datavg dependency = "lv01"
CuVPD: name = "hdisk2" vpd_type = 0 vpd = "*MFIBM *TM\n\ HUS151473VL3800 *F03N5280 *RL53343341*SN009DAFDF*ECH17 923D *P26K5531 *Z0\n\ 000004029F00013A*ZVMPSS43A *Z20068*Z307220"
Figure 2-21. Additional device object classes
AN151.0
Notes: PdCn
The Predefined Connection (PdCn) object class contains connection information for adapters (or sometimes called intermediate devices). This object class also includes predefined dependency information. For each connection location, there are one or more objects describing the subclasses of devices that can be connected. The sample PdCn objects on the visual indicate that, at the given locations, all devices belonging to subclass SCSI could be attached.
CuDep
The Customized Dependency (CuDep) object class describes device instances that depend on other device instances. This object class describes the dependence links between logical devices and physical devices as well as dependence links between

V5.3
Student Notebook
Uempty
logical devices, exclusively. Physical dependencies of one device on another device are recorded in the Customized Devices (CuDev) object class. The sample CuDep objects on the visual show the dependencies between logical volumes and the volume groups they belong to.
CuDvDr
The Customized Device Driver (CuDvDr) object class is used to create the entries in the /dev directory. These special files are used from applications to access a device driver that is part of the AIX kernel. The attribute value1 is called the major number and is a unique key for a device driver. The attribute value2 specifies a certain operating mode of a device driver. The sample CuDvDr objects on the visual reflect the device driver for disk drives hdisk2 and hdisk3. The major number 36 specifies the driver in the kernel. In our example, the minor numbers 0 and 1 specify two different instances of disk dives, both using the same device driver. For other devices, the minor number may represent different modes in which the device can be used. For example, if we were looking at a tape drive, the operating mode 0 would specify a rewind on close for the tape drive, the operating mode 1 would specify no rewind on close for a tape drive.
CuVPD
The Customized Vital Product Data (CuVPD) object class contains vital product data (manufacturer of device, engineering level, part number, and so forth) that is useful for technical support. When an error occurs with a specific device, the vital product data is shown in the error log.

2-39
Student Notebook
Checkpoint
IBM Power Systems
1. In which ODM class do you find the physical volume IDs of your disks?
________________________________________________
2. What is the difference between the states: defined and available?

________________________________________________ ________________________________________________ ________________________________________________ ________________________________________________
AN151.0
Notes:

V5.3
Student Notebook
Uempty
Exercise 3: The Object Data Manager (ODM)

IBM Power Systems
Review of device configuration ODM classes Modifying a device default attribute Creating self-defined ODM classes (Optional)
Figure 2-23. Exercise 3: The Object Data Manager (ODM)
AN151.0
Notes:

2-41
Student Notebook
Unit summary
IBM Power Systems
Having completed this unit, you should be able to: Describe the structure of the ODM Use the ODM command line interface Explain the role of the ODM in device configuration Describe the function of the most important ODM files
AN151.0
Notes:
The ODM is made from object classes, which are broken into individual objects and descriptors. AIX offers a command line interface to work with the ODM files. The device information is held in the customized and the predefined databases (Cu*, Pd*).

V5.3
Student Notebook
Uempty
Unit 3. Error monitoring

This unit covers techniques in monitoring for problems and how to automate responses to those problems. Topics include an overview of the AIX Error Log facility (and how it can interact with the syslogd daemon), the Resource Monitoring and Control (RMC) facility, and the system hang (shdaemon) monitoring facility.

After completing this unit, you should be able to: Analyze error log entries Identify and maintain the error logging components Describe different error notification methods Log system messages using the syslogd daemon Monitor and take actions for threshold conditions using RMC Monitor and take actions for hang conditions using shdaemon

Accountability: Lab exercise Checkpoint questions
References
Online AIX Version 6.1 General Programming Concepts: Writing and Debugging Programs (Chapter 5. Error-Logging Overview) AIX Version 6.1 Command Reference volumes 1-6
Online
Note: References listed as online above are available at the following address: http://publib.boulder.ibm.com/infocenter/systems

3-1
Student Notebook
Unit objectives
IBM Power Systems
After completing this unit, you should be able to: Analyze error log entries Identify and maintain the error logging components Describe different error notification methods Log system messages using the syslogd daemon Monitor and take actions for threshold conditions using RMC Monitor and take actions for hang conditions using shdaemon
AN151.0
Notes:
3-2

V5.3
Student Notebook
Uempty
3.1. Working with the error log

3-3
Student Notebook
Error logging components

IBM Power Systems
console
errnotify
diagnostics error notification
SMIT errpt formatted output
CuDv, CuAt CuVPD error record template /var/adm/ras/errtmplt errstop application errlog() /dev/error (timestamp)
error daemon
errlog /var/adm/ras/errlog /usr/lib/errdemon
errclear errlogger
User Kernel
errsave() kernel module

Figure 3-2. Error logging components
AN151.0
Notes: Detection of an error

The error logging process begins when an operating system module detects an error. The error detecting segment of code then sends error information to either the errsave() kernel service or the errlog() application subroutine, where the information is in turn written to the /dev/error special file. This process then adds a timestamp to the collected data. The errdemon daemon constantly checks the /dev/error file for new entries, and when new data is written, the daemon conducts a series of operations.
Creation of error log entries

Before an entry is written to the error log, the errdemon daemon compares the label sent by the kernel or the application code to the contents of the Error Record Template Repository. If the label matches an item in the repository, the daemon collects additional data from other parts of the system.
3-4

V5.3
Student Notebook
Uempty
To create an entry in the error log, the errdemon daemon retrieves the appropriate template from the repository, the resource name of the unit that caused the error, and the detail data. Also, if the error signifies a hardware-related problem and hardware vital product data (VPD) exists, the daemon retrieves the VPD from the ODM. When you access the error log, either through SMIT or with the errpt command, the error log is formatted according to the error template in the error template repository and presented in either a summary or detailed report. Most entries in the error log are attributable to hardware and software problems, but informational messages can also be logged, for example, by the system administrator.
The errlogger command

The errlogger command allows the system administrator to record messages of up to 1024 bytes in the error log. Whenever you perform a maintenance activity, such as clearing entries from the error log, replacing hardware, or applying a software fix, it is a good idea to record this activity in the system error log. The following example illustrates use of the errlogger command: # errlogger system hard disk (hdisk0) replaced. This message will be listed as part of the error log.
Error log hardening

Under very rare circumstances, such as powering off the system exactly while the errdemon is writing into the error log, the error log may become corrupted. In AIX 5L V5.3, there are minor modifications made to the errdemon to improve its robustness and to recover the error log file at its start. When the errdemon starts, it checks for error log consistency. First, it makes a backup copy of the existing error log file to /tmp/errlog.save, and then it corrects the error log file, while preserving consistent error log entries. The difference from the previous versions of AIX is that the errdemon used to reset the log file if it was corrupted, instead of repairing it.

3-5
Student Notebook
Generating an error report using SMIT

IBM Power Systems
# smit errpt
Generate an Error Report ... CONCURRENT error reporting? Type of Report Error CLASSES (default is all) Error TYPES (default is all) Error LABELS (default is all) Error ID's (default is all) Resource CLASSES (default is all) Resource TYPES (default is all) Resource NAMES (default is all) SEQUENCE numbers (default is all) STARTING time interval ENDING time interval Show only Duplicated Errors Consolidate Duplicated Errors LOGFILE TEMPLATE file MESSAGE file FILENAME to send report to (default is stdout) ...
no summary [] [] [] [] [] [] [] [] [] [] [no] [no] [/var/adm/ras/errlog] [/var/adm/ras/errtmplt] [] []
+ + + + +X
Figure 3-3. Generating an error report using SMIT
AN151.0
Notes: Overview
The SMIT fastpath smit errpt takes you to the screen used to generate an error report. Any user can use this screen. As shown on the visual, the screen includes a number of fields that can be used for report specifications. Some of these fields are described in more detail below.
CONCURRENT error reporting?

Yes means you want errors displayed or printed as the errors are entered into the error log (a sort of tail -f ).
3-6

V5.3
Student Notebook
Uempty
Type of report
Summary, intermediate, and detailed reports are available. Detailed reports give comprehensive information. Intermediate reports display most of the error information. Summary reports contain concise descriptions of errors.
Error classes
Values are H (hardware), S (software), and O (operator messages created with errlogger). You can specify more than one error class.
Error types
Valid error types include the following: - PEND - The loss of availability of a device or component is imminent. - PERF - The performance of the device or component has degraded to below an acceptable level. - TEMP - Recovered from condition after several attempts. - PERM - Unable to recover from error condition. Error types with this value are usually the most severe errors and imply that you have a hardware or software defect. Error types other than PERM usually do not indicate a defect, but they are recorded so that they can be analyzed by the diagnostic programs. - UNKN - Severity of the error cannot be determined. - INFO - The error type is used to record informational entries
Error labels
An error label is the mnemonic name used for an error ID.
Error IDs
An error ID is a 32-bit hexadecimal code used to identify a particular failure.
Resource classes
Means device class for hardware errors (for example, disk).
Resource types
Indicates device type for hardware (for example, 355 MB).

3-7
Student Notebook
Resource names
Provides common device name (for example hdisk0).
Starting and ending time interval

The format mmddhhmmyy can be used to select only errors from the log that are time stamped between the two values.
Show only duplicated errors

Yes will report only those errors that are exact duplicates of previous errors generated during the interval of time specified. The default time interval is 100 milliseconds. This value can be changed with the errdemon -t command. The default for the Show only Duplicated Errors option is no.
Consolidate duplicated errors

Yes will report only the number of duplicate errors and timestamps of the first and last occurrence of that error. The default for the Consolidate Duplicated Errors option is no.
File name to send reports to

The report can be sent to a file. The default is to send the report to stdout.
3-8

V5.3
Student Notebook
Uempty
The errpt command

IBM Power Systems
Summary report:
# errpt
Intermediate report:
# errpt -A
Detailed report:
# errpt -a
Summary report of all hardware errors:

# errpt -d H
Detailed report of all software errors:

# errpt -a -d S
Concurrent error logging ("Real-time" error logging):

# errpt -c > /dev/console
Figure 3-4. The errpt command
AN151.0
Notes: Types of reports available

The errpt command generates a report of logged errors. Three different layouts can be produced, depending on the option that is used: - A summary report gives an overview (default). - An intermediate report only displays the values for the LABEL, Date/Time, Type, Resource Name, Description and Detailed Data fields. Use the option -A to specify an intermediate report. - A detailed report shows a detailed description of all the error entries. Use the option -a to specify a detailed report.

3-9
Student Notebook
The -d option
The -d option (flag) can be used to limit the report to a particular class of errors. Two examples illustrating use of this flag are shown on the visual: - The command errpt -d H specifies a summary report of all hardware (-d H) errors. - The command errpt -a -d S specifies a detailed report (-a) of all software (-d S) errors.
Input file used

The errpt command queries the error log file /var/adm/ras/errlog to produce the error report.
The -c option
If you want to display the error entries concurrently, that is, at the time they are logged, you must execute errpt -c. In the example on the visual, we direct the output to the system console.
The -D flag
Duplicate errors can be consolidated using errpt -D. When used with the -a option, errpt -D reports only the number of duplicate errors and the timestamp for the first and last occurrence of the identical error.
The -P flag
Shows only errors which are duplicates of the previous error. The -P flag applies only to duplicate errors generated by the error log device driver.
Additional information
The errpt command has many options. Refer to your AIX Commands Reference (or the man page for errpt) for a complete description.

V5.3
Student Notebook
Uempty
A summary report (errpt)

IBM Power Systems
# errpt IDENTIFIER 192AC071 C6ACA566 A6DF45AA 2BFA76F6 9DBCFDEE 192AC071 AA8AB241 C6ACA566 2BFA76F6 EAA3D429 EAA3D429 F7DDA124 TIMESTAMP 1010130907 1010130807 1010130707 1010130707 1010130707 1010123907 1010120407 1010120007 1010094907 1010094207 1010094207 1010094207 T U I T T T T U T U U U T O S O S O O O S S S S H C RESOURCE_NAME errdemon syslog RMCdaemon SYSPROC errdemon errdemon OPERATOR syslog SYSPROC LVDD LVDD LVDD DESCRIPTION ERROR LOGGING TURNED OFF MESSAGE REDIRECTED FROM SYSLOG The daemon is started. SYSTEM SHUTDOWN BY USER ERROR LOGGING TURNED ON ERROR LOGGING TURNED OFF OPERATOR NOTIFICATION MESSAGE REDIRECTED FROM SYSLOG SYSTEM SHUTDOWN BY USER PHYSICAL PARTITION MARKED STALE PHYSICAL PARTITION MARKED STALE PHYSICAL VOLUME DECLARED MISSING
Error Type: P: Permanent, Performance, or Pending T: Temporary I: Informational U: Unknown

Figure 3-5. A summary report (errpt)
Error Class: H: Hardware S: Software O: Operator U: Undetermined

AN151.0
Notes: Content of summary report

By default, the errpt command creates a summary report which gives an overview of the different error entries. One line per error is fine to get a feel for what is there, but you need more details to understand problems.
Need for detailed report

The example shows different hardware and software errors that occurred. To get more information about these errors, you must create a detailed report.

3-11
Student Notebook
A detailed error report (errpt -a)

IBM Power Systems
LABEL: IDENTIFIER:
LVM_SA_PVMISS F7DDA124 Wed Oct 10 09:42:20 CDT 2007 113 00C35BA04C00 rt1s3vlp2 H UNKN Global LVDD NONE NONE
Date/Time: Sequence Number: Machine Id: Node Id: Class: Type: WPAR: Resource Name: Resource Class: Resource Type: Location:
Description PHYSICAL VOLUME DECLARED MISSING Probable Causes POWER, DRIVE, ADAPTER, OR CABLE FAILURE Detail Data MAJOR/MINOR DEVICE NUMBER 8000 0011 0000 0001 SENSE DATA 00C3 5BA0 0000 4C00 0000 0115 7F54 BF78 00C3 5BA0 7FCF 6B93 0000 0000 0000 0000
Figure 3-6. A detailed error report (errpt -a)
AN151.0
Notes: Content of detailed error report

As previously mentioned, detailed error reports are generated by issuing the errpt -a command. The first half of the information displayed is obtained from the ODM (CuDv, CuAt, CuVPD) and is very useful because it shows clearly which part causes the error entry. The next few fields explain probable reasons for the problem, and actions that you can take to correct the problem. The last field, SENSE DATA, is a detailed report about which part of the device is failing. For example, with disks, it could tell you which sector on the disk is failing. This information can be used by IBM support to analyze the problem.

V5.3
Student Notebook
Uempty
Interpreting error classes and types

The values shown for error class and error type provide information that is useful in understanding a particular problem: 1. The combination of an error class value of H and an error type value of PERM indicates that the system encountered a problem with a piece of hardware and could not recover from it. 2. The combination of an error class value of H and an error type value of PEND indicates that a piece of hardware may become unavailable soon due to the numerous errors detected by the system. 3. The combination of an error class value of S and an error type of PERM indicates that the system encountered a problem with software and could not recover from it. 4. The combination of an error class value of S and an error type of TEMP indicates that the system encountered a problem with software. After several attempts, the system was able to recover from the problem. 5. An error class value of O indicates that an informational message has been logged. 6. An error class value of U indicates that an error class could not be determined.
Link between error log and diagnostics

In AIX 5L V5.1 and later, there is a link between the error log and diagnostics. Error reports include the diagnostic analysis for errors that have been analyzed. Diagnostics, and the diagnostic tool diag, will be covered in a later unit.

3-13
Student Notebook
Types of disk errors

IBM Power Systems
Error Label DISK_ERR1 DISK_ERR2, DISK_ERR3 DISK_ERR4
Error Recommendations Type P Failure of physical volume media Action: Replace device as soon as possible P T Device does not respond Action: Check power supply Error caused by bad block or occurrence of a recovered error Rule of thumb: If disk produces more than one DISK_ERR4 per week, replace the disk
SCSI_ERR* (SCSI_ERR10) Error Types:
SCSI communication problem Action: Check cable, SCSI addresses, terminator
P = Permanent T = Temporary
Figure 3-7. Types of disk errors
AN151.0
Notes: Common disk errors

The following list explains the most common disk errors you should know about: 1. DISK_ERR1 is caused from wear and tear of the disk. Remove the disk as soon as possible from the system and replace it with a new one. Follow the procedures that you have learned earlier in this course. 2. DISK_ERR2 and DISK_ERR3 error entries are mostly caused by a loss of electrical power. 3. DISK_ERR4 is the most interesting one, and the one that you should watch out for, as this indicates bad blocks on the disk. Do not panic if you get a few entries in the log of this type of an error. What you should be aware of is the number of DISK_ERR4 errors and their frequency. The more you get, the closer you are getting to a disk failure. You want to prevent this before it happens, so monitor the error log closely.

V5.3
Student Notebook
Uempty
4. Sometimes SCSI errors are logged, mostly with the LABEL SCSI_ERR10. They indicate that the SCSI controller is not able to communicate with an attached device. In this case, check the cable (and the cable length), the SCSI addresses, and the terminator.
DISK_ERR5 errors
A very infrequent error is DISK_ERR5. It is the catch-all (that is, the problem does not match any of the above DISK_ERRx symptoms). You need to investigate further by running the diagnostic programs which can detect and produce more information about the problem.

3-15
Student Notebook
LVM error log entries

IBM Power Systems
Error Label LVM_BBEPOOL, LVM_BBERELMAX, LVM_HWFAIL LVM_SA_STALEPP
Class and Type S,P
Recommendations No more bad block relocation Action: Replace disk as soon as possible.
S,P
Stale physical partition Action: Check disk, synchronize data (syncvg).
LVM_SA_QUORCLOSE
H,P
Quorum lost, volume group closing Action: Check disk, consider working without quorum.
Error Classes: H = Hardware S = Software
Error Types: P = Permanent T = Temporary
Figure 3-8. LVM error log entries
AN151.0
Notes: Important LVM error codes

The visual shows some very important LVM error codes you should know. All of these errors are permanent errors that cannot be recovered. Very often these errors are accompanied by hardware errors such as those shown on the previous page.
Immediate response to errors

Errors, such as those shown on the visual, require your immediate intervention.

V5.3
Student Notebook
Uempty
Maintaining the error log

IBM Power Systems
# smit errdemon
Change / Show Characteristics of the Error Log Type or select values in entry fields. Press Enter AFTER making all desired changes. LOGFILE *Maximum LOGSIZE Memory Buffer Size ... [/var/adm/ras/errlog] [1048576] [32768]
# #
# smit errclear
Clean the Error Log Type or select values in entry fields. Press Enter AFTER making all desired changes. Remove entries older than this number of days Error CLASSES Error TYPES ... Resource CLASSES ... [30] [ ] [ ] [ ] + # + +
==> Use the errlogger command as a reminder <==

Figure 3-9. Maintaining the error log
AN151.0
Notes: Changing error log attributes

To change error log attributes like the error log filename, the internal memory buffer size, and the error log file size, use the SMIT fastpath smit errdemon. The error log file is implemented as a ring. When the file reaches its limit, the oldest entry is removed to allow adding a new one. The command that SMIT executes is the errdemon command. See your AIX Commands Reference for a listing of the different options.
Cleaning up error log entries

To clean up error log entries, use the SMIT fastpath smit errclear. For example, after removing a bad disk that caused error logs entries, you should remove the corresponding error log entries regarding the bad disk. The errclear command is part of the fileset bos.sysmgt.serv_aid.

3-17
Student Notebook
Entries in /var/spool/cron/crontabs/root use errclear to remove software and hardware errors. Software and operator errors are purged after 30 days, hardware errors are purged after 90 days.
Using errlogger to create reminders

Follow the suggestion at the bottom of the visual. Whenever an important system event takes place, for example, the replacement of a disk, log this event using the errlogger command.
Full list of characteristics of the error log

The listing shown in the visual is not the complete smit dialogue screen. Following is the complete dialog fields: LOGFILE * Maximum LOGSIZE Memory BUFFER SIZE Duplicate Error Detection Duplicate Time Interval in milliseconds Duplicate error maximum [1000] # [/var/adm/ras/errlog] [1048576] [32768] [true] [10000] # # + #

V5.3
Student Notebook
Uempty
Exercise 9: Error monitoring (part 1)

IBM Power Systems
Part 1: Working with the error log
Figure 3-10. Exercise 2: Error monitoring (part 1)
AN151.0
Notes: Goals for this part of the exercise

The first part of this exercise allows you to work with the AIX error logging facility. After completing this part of the exercise, you should be able to: - Determine what errors are logged on your machine - Generate different error reports - Start concurrent error notification

3-19
Student Notebook

V5.3
Student Notebook
Uempty
3.2. Error notification and syslogd

3-21
Student Notebook
Error notification methods

IBM Power Systems
ODM-Based: /etc/objrepos/errnotify
Error notification
Concurrent Error Logging: errpt -c > /dev/console
Self-made Error Notification
Figure 3-11. Error notification methods
AN151.0
Notes: What is error notification?

Implementing error notification means taking steps that cause the system to inform you whenever an error is posted to the error log.
Ways to implement error notification

There are different ways to implement error notification: 1. Concurrent error logging: This is the easiest way to implement error notification. If you execute errpt -c, each error is reported when it occurs. By redirecting the output to the console, an operator is informed about each new error entry. 2. Self-made error notification: Another easy way to implement error notification is to write a shell procedure that regularly checks the error log. This is illustrated on the next visual.

V5.3
Student Notebook
Uempty
3. ODM-based error notification: The errdemon program uses the ODM class errnotify for error notification. How to work with errnotify is discussed later in this topic.

3-23
Student Notebook
Self-made error notification

IBM Power Systems
#!/usr/bin/ksh errpt > /tmp/errlog.1
while true do sleep 60 errpt > /tmp/errlog.2
# Let's sleep one minute
# Compare the two files. # If no difference, let's sleep again cmp -s /tmp/errlog.1 /tmp/errlog.2 &&
continue
# Files are different: Let's inform the operator: print "Operator: Check error log " > /dev/console errpt done
>
/tmp/errlog.1
Figure 3-12. Self-made error notification
AN151.0
Notes: Implementing self-made error notification

It is very easy to implement self-made error notification by using the errpt command. The sample shell script on the visual shows how this can be done.
Discussion of example on visual

The procedure on the visual shows a very easy but effective way of implementing error notification. Let's analyze this procedure: - The first errpt command generates a file /tmp/errlog.1. - The construct while true implements an infinite loop that never terminates. - In the loop, the first action is to sleep one minute. - The second errpt command generates a second file /tmp/errlog.2.

V5.3
Student Notebook
Uempty
- The two files are compared using the command cmp -s (silent compare, that means no output will be reported). If the files are not different, we jump back to the beginning of the loop (continue), and the process will sleep again. - If there is a difference, a new error entry has been posted to the error log. In this case, we inform the operator that a new entry is in the error log. Instead of print you could use the mail command to inform another person.

3-25
Student Notebook
ODM-based error notification: errnotify

IBM Power Systems
errnotify: en_pid = 0 en_name = "sample" en_persistenceflg = 1 en_label = "" en_crcid = 0 en_class = "H" en_type = "PERM" en_alertflg = "" en_resource = "" en_rtype = "" en_rclass = "disk" en_method = "errpt -a -l $1 | mail -s DiskError root"
Figure 3-13. ODM-based error notification: errnotify
AN151.0
Notes: The error notification object class

The Error Notification object class specifies the conditions and actions to be taken when errors are recorded in the system error log. The user specifies these conditions and actions in an Error Notification object. Each time an error is logged, the error notification daemon determines if the error log entry matches the selection criteria of any of the Error Notification objects. If matches exist, the daemon runs the programmed action, also called a notify method, for each matched object. The Error Notification object class is located in the /etc/objrepos/errnotify file. Error Notification objects are added to the object class by using ODM commands.

V5.3
Student Notebook
Uempty
Example on visual
The example on the visual shows an object that creates a mail message to root whenever a disk error is posted to the log.
List of descriptors
Here is a list of all descriptors for the errnotify object class: en_alertflg Identifies whether the error is alertable. This descriptor is provided for use by alert agents with network management applications. The values are TRUE (alertable) or FALSE (not alertable). Identifies the class of error log entries to match. Valid values are H (hardware errors), S (software errors), O (operator messages), and U (undetermined). Specifies the error identifier associated with a particular error. Specifies the label associated with a particular error identifier as defined in the output of errpt -t (show templates). Specifies a user-programmable action, such as a shell script or a command string, to be run when an error matching the selection criteria of this Error Notification object is logged. The error notification daemon uses the sh -c command to execute the notify method. The following keywords are passed to the method as arguments: $1 Sequence number from the error log entry $2 Error ID from the error log entry $3 Class from the error log entry $4 Type from the error log entry $5 Alert flags from the error log entry $6 Resource name from the error log entry $7 Resource type from the error log entry $8 Resource class from the error log entry $9 Error label from the error log entry en_name Uniquely identifies the object
en_class
en_crcid en_label en_method
en_persistenceflg Designates whether the Error Notification object should be removed when the system is restarted. 0 means removed at boot time; 1 means persists through boot.

3-27
Student Notebook
en_pid
Specifies a process ID for use in identifying the Error Notification object. Objects that have a PID specified should have the en_persistenceflg descriptor set to 0. Identifies the class of the failing resource. For hardware errors, the resource class is the device class (see PdDv). Not used for software errors. Identifies the name of the failing resource. For hardware errors, the resource name is the device name. Not used for software errors. Identifies the type of the failing resource. For hardware errors, the resource type is the device type (see PdDv). Not used for software errors. Enables notification of an error accompanied by a symptom string when set to TRUE. Identifies the severity of error log entries to match. Valid values are: INFO: Informational PEND: Impending loss of availability PERM: Permanent PERF: Unacceptable performance degradation TEMP: Temporary UNKN: Unknown TRUE: Matches alertable errors FALSE: Matches non-alertable errors 0: Removes the Error Notification object at system restart non-zero: Retains the Error Notification object at system restart
en_rclass
en_resource
en_rtype
en_symptom en_type
en_err64 en_dup
Identifies the environment of the error. TRUE indicates that the error is from a 64-bit environment. Identifies whether the kernel identified the error as a duplicate. TRUE indicates that it is a duplicate error.

V5.3
Student Notebook
Uempty
syslogd daemon
IBM Power Systems
/etc/syslog.conf: daemon.debug /tmp/syslog.debug
/tmp/syslog.debug:
syslogd
inetd[16634]: A connection requires tn service inetd[16634]: Child process 17212 has ended
# stopsrc
-s
inetd inetd -a "-d"
# startsrc -s
Provide debug information.
Figure 3-14. syslogd daemon
AN151.0
Notes: Function of syslogd

The syslogd daemon logs system messages from different software components (kernel, daemon processes, system applications).
The /etc/syslog.conf configuration file

When started, the syslogd reads a configuration file /etc/syslog.conf. Whenever you change this configuration file, you need to refresh the syslogd subsystem: # refresh -s syslogd

3-29
Student Notebook
Discussion of example on visual

The visual shows a configuration that is often used when a daemon process causes a problem. The following line is placed in /etc/syslog.conf and indicates that facility daemon should be monitored/controlled: daemon.debug /tmp/syslog.debug The line shown also specifies that all messages with the priority level debug and higher, should be written to the file /tmp/syslog.debug. Note that this file must exist. The daemon process that causes problems (in our example the inetd) is started with option -d to provide debug information. This debug information is collected by the syslogd daemon, which writes the information to the log file /tmp/syslog.debug.

V5.3
Student Notebook
Uempty
syslogd configuration examples

IBM Power Systems
/etc/syslog.conf: auth.debug /dev/console

All security messages to the system console Collect all mail messages in /tmp/mail.debug Collect all daemon messages in /tmp/daemon.debug Send all messages, except mail messages, to host server
mail.debug
/tmp/mail.debug
daemon.debug /tmp/daemon.debug
*.debug; mail.none
@server
After changing /etc/syslog.conf: # refresh -s syslogd
Figure 3-15. syslogd configuration examples
AN151.0
Notes: Discussion of examples on visual

The visual shows some examples of syslogd configuration entries that might be placed in /etc/syslog.conf: - The following line specifies that all security messages are to be directed to the system console: auth.debug /dev/console
- The following line specifies that all mail messages are to be collected in the file /tmp/mail.debug: mail.debug /dev/mail.debug
- The following line specifies that all messages produced from daemon processes are to be collected in the file /tmp/daemon.debug: daemon.debug
/tmp/daemon.debug
3-31
Student Notebook
- The following line specifies that all messages, except messages from the mail subsystem, are to be sent to the syslogd daemon on the host server: *.debug; mail.none @server
Note that, if this example and the preceding example appear in the same /etc/syslog.conf file, messages sent to /tmp/daemon.debug will also be sent to the host server.
General format of /etc/syslog.conf entries

As you see, the general format for entries in /etc/syslog.conf is: selector action The selector field names a facility and a priority level. Separate facility names with a comma (,). Separate the facility and priority level portions of the selector field with a period (.). Separate multiple entries in the same selector field with a semicolon (;). To select all facilities use an asterisk (*). The action field identifies a destination (file, host or user) to receive the messages. If routed to a remote host, the remote system will handle the message as indicated in its own configuration file. To display messages on a user's terminal, the destination field must contain the name of a valid, logged-in system user. If you specify an asterisk (*) in the action field, a message is sent to all logged-in users.
Facilities
Use the following system facility names in the selector field: kern user mail daemon auth syslog lpr news uucp * Kernel User level Mail subsystem System daemons Security or authorization syslogd messages Line-printer subsystem News subsystem uucp subsystem All facilities
Priority levels
Use the following levels in the selector field. Messages of the specified level and all levels above it are sent as directed.
V5.3
Student Notebook
Uempty
emerg alert crit err warning notice info debug none
Specifies emergency messages. These messages are not distributed to all users. Specifies important messages such as serious hardware errors. These messages are distributed to all users. Specifies critical messages, not classified as errors, such as improper login attempts. These messages are sent to the system console. Specifies messages that represent error conditions. Specifies messages for abnormal, but recoverable conditions. Specifies important informational messages. Specifies information messages that are useful in analyzing the system. Specifies debugging messages. If you are interested in all messages of a certain facility, use this level. Excludes the selected facility.
Refreshing the syslogd subsystem

As previously mentioned, after changing /etc/syslog.conf, you must refresh the syslogd subsystem in order to have the change take effect. Use the following command to accomplish this: # refresh -s syslogd

3-33
Student Notebook
Redirecting syslog messages to error log

IBM Power Systems
/etc/syslog.conf: *.debug errlog
Redirect all syslog messages to error log
# errpt IDENTIFIER TIMESTAMP T C ... C6ACA566 0505071399 U FROM SYSLOG ... RESOURCE_NAME DESCRIPTION S syslog MESSAGE REDIRECTED
Figure 3-16. Redirecting syslog messages to error log
AN151.0
Notes: Consolidating error messages

Some applications use syslogd for logging errors and events. Some administrators find it desirable to list all errors in one report.
Redirecting messages from syslogd to the error log

The visual shows how to redirect messages from syslogd to the error log. By setting the action field to errlog, all messages are redirected to the AIX error log.

V5.3
Student Notebook
Uempty
Directing error log messages to syslogd

IBM Power Systems
errnotify:
en_name = "syslog1" en_persistenceflg = l en_method = "logger Error Log: `errpt -l $1 | grep -v TIMESTAMP`"
errnotify:
en_name = "syslog1" en_persistenceflg = l en_method = "logger Error Log: $(errpt -l $1 | grep -v TIMESTAMP)"
Direct the last error entry (-l $1) to the syslogd. Do not show the error log header (grep -v) or (tail -1).
errnotify:
en_name = "syslog1" en_persistenceflg = l en_method = "errpt -l $1 | tail -1 | logger -t errpt -p daemon.notice"
Figure 3-17. Directing error log messages to syslogd
AN151.0
Notes: Using the logger command

You can direct error log events to syslogd by using the logger command with the errnotify ODM class. Using objects such as those shown on the visual, whenever an entry is posted to the error log, this last entry can be passed to the logger command.
Command substitution
You will need to use command substitution (or pipes) before calling the logger command. The first two examples on the visual illustrate the two ways to do command substitution in a Korn shell environment: - Using the UNIX command syntax (with backquotes) - shown in the first example on the visual - Using the newer $(UNIX command) syntax - shown in the second example on the visual
Copyright IBM Corp. 2009 Unit 3. Error monitoring
3-35
Student Notebook
System hang detection

IBM Power Systems
System hangs:
High priority process Other
What does shdaemon do?

Monitors system's ability to run processes Takes specified action if threshold is crossed
Actions:
Log error in the Error log Display a warning message on the console Launch recovery login on a console Launch a command Automatically REBOOT system
Figure 3-18. System hang detection
AN151.0
Notes: Types of system hangs

shdaemon can help recover from certain types of system hangs. For our purposes, we will divide system hangs into two types: - High priority process The system may appear to be hung if some applications have adjusted their process or thread priorities so high that regular processes are not scheduled. In this case, work is still being done, but only by the high priority processes. As currently implemented, shdaemon specifically addresses this type of hang. - Other Other types of hangs may be caused by a variety of problems. For example, system thrashing, kernel deadlock, and the kernel in tight loop. In these cases, no (or very little) meaningful work will get done. shdaemon may help with some of these problems.
V5.3
Student Notebook
Uempty
What does shdaemon do?

If enabled, shdaemon monitors the system to see if any process with a process priority number, higher than a set threshold, has been run during a set time-out period. Remember that a higher process priority number indicates a lower priority on the system. In effect, shdaemon monitors to see if lower priority processes are being scheduled. shdaemon runs at the highest priority (priority number = 0), so that it will always be able to get CPU time, even if a process is running at very high priority.
Actions
If lower priority processes are not being scheduled, shdaemon will perform the specified action. Each action can be individually enabled and has its own configurable priority and time-out values. There are five actions available: - Log error in the Error log. - Display a warning message on a console. - Launch a recovery login on a console. - Launch a command. - Automatically REBOOT the system.

3-37
Student Notebook
Configuring shdaemon
IBM Power Systems
# shconf -E -l prio sh_pp disable pp_errlog pp_eto pp_eprio pp_warning pp_wto pp_wprio pp_wterm pp_login pp_lto pp_lprio pp_lterm pp_cmd pp_cto pp_cprio pp_cpath pp_reboot pp_rto pp_rprio disable 2 60 enable 2 60 /dev/console enable 2 100 /dev/console
Enable Process Priority Problem Log Error in the Error Logging Detection Time-out Process Priority Display a warning message on a console Detection Time-out Process Priority Terminal Device Launch a recovering login on a console Detection Time-out Process Priority Terminal Device
disable Launch a command 2 Detection Time-out 60 Process Priority /home/unhang Script disable 5 39 Automatically REBOOT system Detection Time-out Process Priority
Figure 3-19. Configuring shdaemon
AN151.0
Notes: Introduction
shdaemon configuration information is stored as attributes in the SWservAt ODM object class. Configuration changes take effect immediately and survive across reboots. Use shconf (or smit shd) to configure or display the current configuration of shdaemon. The values shown in the visual are the default values.
Enabling shdaemon
At least two parameters must be modified to enable shdaemon: - Enable priority monitoring (sh_pp) - Enable one or more actions (pp_errlog, pp_warning, and so forth)

V5.3
Student Notebook
Uempty
When enabling shdaemon, shconf performs the following steps: - Modifies the SWservAt parameters - Starts shdaemon - Modifies /etc/inittab so that shdaemon will be started on each system boot
Action attributes
Each action has its own attributes, which set the priority and timeout thresholds and define the action to be taken. The timeout attribute unit of measure is in minutes.
Example
By changing the chconf attributes, we can enable, disable, and modify the behavior of the facility. For example:, shdaemon is enabled to monitor process priority (sh_pp=enable), and the following actions are enabled: - Enable the to monitor process priority monitoring: # shconf -l prio -a sh_pp=enable - Log error in the Error Logging: # shconf -l prio -a pp_errlog=enable Every two minutes (pp_eto=2), shdaemon will check to see if any process has been run with a process priority number greater than 60 (pp_eprio=60). If not, shdaemon logs an error to the error log. - Display a warning message on a console: # shconf -l prio -a pp_warning=enable (default value)
Every two minutes (pp_wto=2), shdaemon will check to see if any process has been run with a process priority number greater than 60 (pp_wprio=60). If not, shdaemon sends a warning message to the console specified by pp_wterm. - Launch a command: # shconf -l prio -a pp_cmd=enable -a pp_cto=5 Every five minutes (pp_cto=5), shdaemon will check to see if any process has been run with a process priority number greater than 60 (pp_cprio=60). If not, shdaemon runs the command specified by pp_cpath (in this case, /home/unhang).

3-39
Student Notebook

IBM Power Systems
Part 2, section 1: Working with syslogd Part 2, section 2: Error notification with errnotify
AN151.0
Notes:

V5.3
Student Notebook
Uempty
3.3. Resource monitoring and control

3-41
Student Notebook
Resource monitoring and control (RMC)

IBM Power Systems
Based on two concepts:

Conditions Responses
Associates predefined responses with predefined conditions for monitoring system resources Example: Broadcast a message to the system administrator when the /tmp file system becomes 90% full
Figure 3-21. Resource monitoring and control (RMC)
AN151.0
Notes: Resource monitoring and control (RMC) basics

RMC is automatically installed and configured when AIX is installed. RMC is started by an entry in /etc/inittab: ctrmc:2:once:/usr/bin/startsrc -s ctrmc > /dev/console 2>&1 To provide a ready-to-use system, 84 conditions, 8 responses are predefined. You can: - Use them as they are - Customize them - Use as templates to define your own To monitor a condition, simply associate one or more responses with the condition. A log file is maintained in /var/ct.

V5.3
Student Notebook
Uempty
Set up
The following steps are provided to assist you in setting up an efficient monitoring system: 1. Review the predefined conditions of your interests. Use them as they are, customize them to fit your configurations, or use them as templates to create your own. 2. Review the predefined responses. Customize them to suit your environment and your working schedule. For example, the response Critical notifications is predefined with three actions: a) Log events to /tmp/criticalEvents. b) E-mail to root. c) Broadcast a message to all logged-in users anytime when an event or a rearm event occurs. You may modify the response, such as to log events to a different file anytime when events occur, e-mail to you during non-working hours, and add a new action to page you only during working hours. With such a setup, different notification mechanisms can be automatically switched, based on your working schedule. 3. Reuse the responses for conditions. For example, you can customize the three severity responses, Critical notifications, Warning notifications, and Informational notifications to take actions in response to events of different severities, and associate the responses to the conditions of respective severities. With only three notification responses, you can be notified of all the events with respective notification mechanisms based on their urgencies. 4. Once the monitoring is set up, your system continues being monitored whether your Web-based System Manager session is running or not. To know the system status, you may bring up a Web-based System Manager session and view the Events plug-in, or simply use the lsaudrec command from the command line interface to view the audit log.
More information
A very good Redbook describing this topic is: A Practical Guide for Resource Monitoring and Control (SG24-6615). This redbook can be found at http://www.redbooks.ibm.com/redbooks/pdfs/sg246615.pdf.

3-43
Student Notebook
RMC conditions property screen: General tab

IBM Power Systems
Figure 3-22. RMC conditions property screen: General tab
AN151.0
Notes: Conditions
A condition monitors a specific property, such as total percentage used, in a specific resource class, such as JFS. Each condition contains an event expression to define an event and an optional rearm event.

V5.3
Student Notebook
Uempty
RMC conditions property screen: Monitored Resources tab

IBM Power Systems
Figure 3-23. RMC conditions property screen: Monitored Resources tab
AN151.0
Notes: Monitoring condition

You can monitor the condition for one or more resources within the monitored property, such as /tmp, or /tmp and /var, or all of the file systems.

3-45
Student Notebook
RMC actions property screen: General tab

IBM Power Systems
Figure 3-24. RMC actions property screen: General tab
AN151.0
Notes: Defining an action

To define an action, you can choose one of the following predefined commands: - Send mail - Log an entry to a file - Broadcast a message - Send an SNMP trap You can also specify an arbitrary program or script of your own by using the Run program option.

V5.3
Student Notebook
Uempty
RMC actions property screen: When in Effect tab

IBM Power Systems
Figure 3-25. RMC actions property screen: When in Effect tab
AN151.0
Notes: When is an event active?

The action can be active for an event only, for a rearm event only, or for both. You can also specify a time window in which the action is active, such as always, or only during on-shift on weekdays. Once the monitoring is set up, the system continues to be monitored whether a Web-based System Manager session is running or not.

3-47
Student Notebook
RMC management
IBM Power Systems
Resource Monitoring and Control (RMC) daemons

Started from /etc/inittab Subsystem name is ctrmc Run in both the partition and on the HMC
To list the status of the RMC daemons:

# lssrc a | grep rsct
To stop the daemons (LPAR)

# /usr/sbin/rsct/bin/rmcctrl z
To start the daemons (LPAR) and enable remote client communications

# /usr/sbin/rsct/bin/rmcctrl A # /usr/sbin/rsct/bin/rmcctrl p
RMC also supports coordination of systems in a cluster

Used by the HMC for service tools and for dynamic LPAR operations
Figure 3-26. RMC management
AN151.0
Notes: Verifying RMC daemons on the AIX partitions

The Resource Monitoring and Control (RMC) daemons are part of the Reliable, Scalable Cluster Technology (RSCT) and are controlled by the System Resource Controller. These daemons run in all partitions and communicate with equivalent RMC daemons running on the HMC. The daemons start automatically when the operating system starts and synchronize with the HMC RMC daemons.
What RMC daemons should be running?

Some daemons will start and stop as needed; so do not be too concerned if your favorite one is not showing at any particular moment. Some may even show as inactive which is fine; they become active when needed. You should, however, see some running.

V5.3
Student Notebook
Uempty
Log in as root to use lssrc -a

If you are not logged in as root when you run this command you will see the error message: The System Resource Controller is having socket problems.
Stopping and starting the RMC daemons

Normally, you should not have to stop and restart the daemons. They are started from /etc/inittab and should work out of the box. If you cannot find any other obvious issues, you can try stopping and starting the RMC daemons. To stop the daemons: /usr/sbin/rsct/bin/rmcctrl -z To start the daemons: /usr/sbin/rsct/bin/rmcctrl -A To enable the daemons for remote client connections (HMC to LPAR and vice versa): /usr/sbin/rsct/bin/rmcctrl -p If you are familiar with the System Resource Controller (SRC) you might be tempted to use stopsrc and startsrc commands to stop and start these daemons. Do not do it; use the rmcctrl commands instead.

3-49
Student Notebook

IBM Power Systems
Part 3: Resource Monitoring
AN151.0
Notes: Goals for this part of the exercise

After completing this part of the exercise, you should be able to: - Define a condition and an action to take when the event occurs.

V5.3
Student Notebook
Uempty
Checkpoint
IBM Power Systems
1. Which command generates error reports? Which flag of this command is used to generate a detailed error report?
__________________________________________________ __________________________________________________
2. Which type of disk error indicates bad blocks?

__________________________________________________
3. What do the following commands do?

errclear _________________________________________ errlogger *.debug errlog __________________________________________________ _________________________________________
4. What does the following line in /etc/syslog.conf indicate?
5. What does the descriptor en_method in errnotify indicate?

___________________________________________________ ___________________________________________________ ___________________________________________________
AN151.0
Notes:

3-51
Student Notebook
Unit summary
IBM Power Systems
Having completed this unit, you should be able to: Analyze error log entries Identify and maintain the error logging components Describe different error notification methods Log system messages using the syslogd daemon Monitor and take actions for threshold conditions using RMC Monitor and take actions for hang conditions using shdaemon
AN151.0
Notes:
Use the errpt (smit errpt) command to generate error reports. Different error notification methods are available. Use smit errdemon and smit errclear to maintain the error log. Some components use syslogd for error logging. The syslogd configuration file is /etc/syslog.conf. You can redirect syslogd and error log messages. You can monitor resource conditions and take automated action, such as sending mail to root.

V5.3
Student Notebook
Uempty
Unit 4. Network Installation Manager basics

This unit provides an introduction to using the Network Installation Manager (NIM) to network boot an AIX client system. It covers the basic installation and configuration of NIM for supporting client installation or booting to maintenance mode.

After completing this unit, you should be able to: Configure an AIX partition for use as a NIM master Set up NIM to support the installation of AIX onto a client

Accountability: Checkpoint Machine exercises
References
SC23-6616 SG24-7296 AIX Version 6.1 Installation and migration NIM from A to Z in AIX 5L (Redbook)
http://www.redbooks.ibm.com IBM Redbooks

4-1
Student Notebook
Unit objectives
IBM Power Systems
After completing this unit, you should be able to: Configure an AIX partition for use as a NIM master Set up NIM to support the installation of AIX onto a client
AN151.0
Notes:
4-2

V5.3
Student Notebook
Uempty
NIM overview
IBM Power Systems
AIX software administration over the network:

Install Update Maintain
Eliminate tape/CD at each system Distribute installation load Support for push or pull installations NIM administrative tools
Command line interface SMIT WebSM
NIM master and NIM server
PUSH installation: Initiated by master
PULL installation: Requested by client
Client and NIM server

Client
Client
Figure 4-2. NIM overview
AN151.0
Notes: Purpose of NIM

NIM provides centralized AIX software administration for multiple machines over the network. NIM supports full AIX operating system installation as well as installing or updating individual packages and performing software maintenance.
Advantages
NIM provides several advantages: - Provides one central point for AIX software administration for all the NIM clients - Eliminates need to walk a CDROM or tape to each system and the need for a tape drive or CDROM drive at every system - Installations can be initiated from the master machine (push) or from the client (pull)

4-3
Student Notebook
- The installation load can be distributed. Most simply, the NIM master machine is configured as the server for all the filesets to be installed. However, you can also configure one or more client machines to act as servers to distribute the load if you have many clients.
NIM administrative tools

There are several different ways you can manage your NIM environment: Method Command Line Description The command line gives you complete control, but the number of options needed can be somewhat daunting. Still, if you want to script NIM operations, you must use the command line. The basic NIM commands are: nimconfig: Configure NIM master. nim: Perform NIM operations from the master. nimclient: Perform NIM operations from a client. niminit: Configure NIM client. lsnim: List information about NIM objects. There are basically two paths into SMITs NIM interface: smit nim: Configure master and client machines and perform all NIM operations. smit eznim: This provides a simplified environment to configure machines and perform some basic NIM operations. This may be a good starting point for a new NIM system administrator. You can also used IBMs Web-based System Manager to configure and manage your NIM environment.
SMIT
Web-based System Manager (wsm)
- As you become familiar with the NIM environment, you may find that you use a combination of methods. For example, you may use the command line to list NIM status and perform simple NIM operations, while using SMIT or WebSM for more complex operations or for operations that you do not perform frequently.
4-4

V5.3
Student Notebook
Uempty
Machine roles
IBM Power Systems
Master
File sets:
bos.sysmgt.nim.master bos.sysmgt.nim.client Stores NIM database
NIM administration Can initiate push installations to NIM clients AIX version >= all other NIM machines
Client
File sets:
bos.sysmgt.nim.client
Can initiate pull installations from a server
Server
Any machine, master or client Serves NIM resources to clients, thus requires adequate disk space and throughput
Figure 4-3. Machine roles
AN151.0
Notes:
There are three basic roles that a machine can assume in the NIM environment: master, client, and resource server. There can only be one master machine in a NIM environment, all other machines are clients. Any machine, master or client, can be a resource server.
NIM software
All machines in the NIM environment must install bos.sysmgt.nim.client. The master machine must also install bos.sysmgt.nim.master and bos.sysmgt.nim.spot.
Master
The NIM master manages all other machines that participate in the NIM environment. The NIM database is stored on the NIM master. The NIM master is fundamental for all

4-5
Student Notebook
of the operations in the NIM environment and must be set up and operational before performing any NIM operations. The master can initiate a software installation to a client, which is called a push installation. Also, the NIM master is the only machine that is given the permissions and ability to execute NIM operations on other machines within the NIM environment. The rsh command is used to remotely execute commands on clients which allows the NIM master to install to a number of clients with one NIM operation. With AIX 5.3 or AIX 6.1, nimsh can be used as an alternative to rsh.
Client
All other machines in a NIM environment are clients. Clients can request a software installation from a server machine (pull installation).
Server
Any machine, the master or a client, can be configured by the master as a server for a particular software resource. Most often, the master is also the server. However, if your environment has many nodes or consists of a complex network environment, you may want to configure some nodes to act as servers to improve installation performance. Servers must have adequate disk space for the resources they will be providing. They also need network connections to the client machines they serve and sufficient bandwidth to respond to the expected volume.
4-6


PowerSystems For AIX-III

Загружено:

Сведения о документе

Оригинальное название

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

PowerSystems For AIX-III

Загружено:

Авторское право:

Доступные форматы

V5.

November 2009 edition

Copyright IBM Corp. 2009

Copyright IBM Corp. 2009

6-1 6-2 6-3 6-4 6-6

Copyright IBM Corp. 2009

7-47 7-49 7-51 7-53 7-55 7-56 7-57

Copyright IBM Corp. 2009

Copyright IBM Corp. 2009

AIX Advanced Administration

Copyright IBM Corp. 2009

Copyright IBM Corp. 2009

AIX Advanced Administration

Copyright IBM Corp. 2009

Copyright IBM Corp. 2009

AIX Advanced Administration

Copyright IBM Corp. 2009

Copyright IBM Corp. 2009

AIX Advanced Administration

Copyright IBM Corp. 2009

xviii AIX Advanced Administration

Copyright IBM Corp. 2009

Unit 1. Advanced AIX administration overview

What you should be able to do

How you will check your progress

Copyright IBM Corp. 2009

Unit 1. Advanced AIX administration overview

Copyright IBM Corporation 2009

Figure 1-1. Unit objectives

AIX Advanced Administration

Copyright IBM Corp. 2009

Functional or performance Avoid unplanned outages with best practices

Avoid planned outages

Use maintenance windows

Effective problem determination and recovery

Figure 1-2. Application outages

AIX Advanced Administration

Copyright IBM Corp. 2009

Live Partition Mobility versus Live Application Mobility

Copyright IBM Corporation 2009

Figure 1-3. Live Partition Mobility versus Live Application Mobility

Live Partition Mobility

Live Application Mobility

Copyright IBM Corp. 2009

Unit 1. Advanced AIX administration overview

AIX Advanced Administration

Copyright IBM Corp. 2009

Maintenance window tasks

Minimize time needed for tasks Operating system maintenance

Copyright IBM Corporation 2009

Figure 1-4. Maintenance window tasks

Notes: Expediting work in the maintenance window

Operating system maintenance

Copyright IBM Corp. 2009

Unit 1. Advanced AIX administration overview

AIX Advanced Administration

Copyright IBM Corp. 2009

Copyright IBM Corp. 2009

Unit 1. Advanced AIX administration overview

Effective problem management

Copyright IBM Corporation 2009

Figure 1-5. Effective problem management

Notes: Obtaining and documenting information about your system