Automatic Storage Management

(ASM)

Speaker Introduction

Michael Messina Senior Managing Consultant Rolta, TUSC Infrastructure Services Working with Oracle Approximately 17 years Background includes Performance Tuning, High Availability and Disaster Recovery Oracle Database OCP Oracle ACE Mike.Messina@roltasolutions.com www.tusc.com / www.roltasolutions.com

Contents

Automatic Storage Management (ASM)

Overview ASM Structure / Components Benefits of ASM Any Issues? Storage Options Redundancy
External, Normal, High Multiple Paths to Storage

Infrastructure

Database Storage File Storage (ASM Clustered File System)

Contents

Volume Management

Comparison to other Volume Managers Use with Other Volume Managers How ASM can replace other Logical Volume Managers Fail Groups Real Application Clusters (RAC) Database DataGuard Requirements ASMLib for Linux Create disks for ASM

ASM in a High Availability Environment

Prepare Volumes for ASM

Contents

Installing ASM

Software and OS Requirements for Oracle Enterprise Linux Installation Basics What is the ASM Command Line Utility ASMCMD Commands

ASM Command Line Utility

Contents

Working with ASM Disk Groups

Discovering Disks Mounting/Un-mounting Disk Groups Create ASM Disk Groups Add Disk/Space to ASM Disk Group Remove Disk/Space from ASM Disk Group Change ASM Disk Group Compatibility Rename Disk Group Managing ASM File Access Control for Disk Groups Drop ASM Disk Group

Contents

ASM Cluster File System

Overview

Oracle ACFS Driver Model Oracle ACFS Mount Model and Namespace Oracle ACFS Mount Registry Oracle ACFS Snapshots Oracle ACFS and Backup and Restore Oracle ACFS Integration with ASM

Loading ACFS Drivers / Modules Create/Manage ACFS File System

AUTOMATIC STORAGE MANAGEMENT (ASM)

Overview

A Volume Manager for Databases A File System for Databases 11gR2 a File System 11gR2 a Cluster File System

ASM Structure / Components

ASM Instance ASM Disk Groups

ASM Disks
ASM Files

Control files Data files, temporary data files, and data file copies SPFILEs Online redo logs, archive logs, and Flashback logs RMAN backups Disaster recovery configurations Change tracking bitmaps Data Pump dumpsets

ASM Structure / Components

January 12 / Slide 11 / Title of Document

Benefits of ASM

Dynamically Add and Remove Disk/Space while database is running Works with Oracle Managed Files (OMF) ASM can coexist with other storage types ASM Can Improve I/O Performance When in a Cluster ASM Supports Rolling Upgrades ASM Reduces Management and Administrative Overhead

ASM can support storage for multiple Databases and Database Versions
Flexible Redundancy Options Oracle Enterprise Manager Integration

January 12 / Slide 12 / Title of Document

Any Issues?

ASM Requires additional Home location for single instance implementations, meaning additional space for install is required. For Normal or High Redundancy Disk Groups the disk for the failgroup(s) are required when the disk group is created. RMAN is required to backup databases using ASM. ASM does not provide any power management features, all power management features are controlled outside the ASM technology stack.

January 12 / Slide 13 / Title of Document

INFRASTRUCTURE

January 12 / Slide 14 / Title of Document

Storage Options

ASM works with virtually any storage type

Direct Attached Storage Network Attached Storage Storage Area Networks Solid State Disk

Storage regardless of the type is presented to the server as a RAW device(s) Operating System on the Server Controls disk use

January 12 / Slide 15 / Title of Document

Redundancy

External

Disk hardware controls the Redundancy 1 disk mirror or failgroup 2 disk mirrors or failgroups

Normal

High

January 12 / Slide 16 / Title of Document

Redundancy

Multipathing

provide failover by using redundant physical paths between server and the storage subsystem. If one or more components fails, then applications can still access data on the storage

Multipathing used with Storage Area Network (SAN), Host Bus Adapter, interface cable, or host port on a multiported storage array.

January 12 / Slide 17 / Title of Document

Database Storage

Disks assigned to ASM Disk Groups ASM Instance manages database storage to disk groups

Database datafiles, tempfiles, controlfiles, spfiles and redo logs stored in ASM Disk Groups
Database can store files in one or more Disk Groups

January 12 / Slide 18 / Title of Document

File Storage (ASM Clustered File System)

ASM Clustered File System (ACFS) 11gR2 added the capability for ASM to act as a file system for all operating system files with ACFS Can act as a local filesystem or a cluster filesystem Works natively at the OS layer to act just like any other filesystem ASM Creation Assistant can be used to create ACFS filesystems ASM command line utility can be used to create and manage ACFS file systems (asmcmd)
January 12 / Slide 19 / Title of Document

VOLUME MANAGEMENT

January 12 / Slide 20 / Title of Document

Comparison to other Volume Managers

ASM Requires RMAN for Database Backups while other volume managers will allow non-RMAN backups.

ASM is being taught with all Oracle Database Training and it Oracles Direction for the future. Other volume managers may require additional training to utilize effectively.
ASM Cluster File System adds capability to utilize ASM as a regular native OS filesystem. ASM is a no-cost cluster file system option, 3rd party volume managers require additional license purchase and support costs. ASM typically provides better performance

Use with Other Volume Managers

ASM co-exists with other cluster file system or native OS file systems.

Mix database and non-database files on different filesystems

Use ASM for Database files and non-database files Use OCFS2, Veritas, ext3, etc. database and non database files Use ASM, OCFS2, Veritas, etc. for database and non-database files

Move database and non-database files between different filesystems including ASM.

How ASM can replace other Logical Volume Managers

11gR2 adds ASM Cluster File System expanding the capability of ASM to be used as a native OS filesystem.

Create ASM file systems/volumes for databases Create ACFS file systems which use ASM for storage for nondatabase files.

ACFS provides full volume management for virtually all file types. ASM with ACFS however can not replace the native OS volume manager for root and boot volumes and ASM/ACFS was not designed to support root or boot volumes for the operating system but to supplement and add additional capabilities

ASM IN A HIGH AVAILABILITY ENVIRONMENT

Fail Groups

FailGroups are utilized in NORMAL and HIGH Redundancy Disk Groups.

FailGroups provide Disk Group storage protection using mirrors within Disk Groups
FailGroups can be located on multiple storage systems protecting against storage system failure Failgroups can be located at a separate site based on network and latency conditions.

Real Application Clusters (RAC) Database

ASM was designed to provide a Highly Available Clustered Storage for Real Application Clusters

Provide highest availability for RAC

External storage Redundancy using RAID 10, RAID 5, RAID 6 to protect against individual disk failures ASM FailGroups to protect against Storage System Failures, storage systems can be located at separate sites depending on network capability and latency.

Multiple Nodes in cluster provide protection against node failure.

DataGuard

Data Guard Provides for long distance cross site failover Uses ASM effectively

ASM can simplify the building and maintaining Data Guard Standby database
RAC capable standbys from RAC primary or single instance standby from RAC primary. Closes the maximum availability and scalability gap that RAC and ASM alone can not provide.

PREPARE VOLUMES FOR ASM

Requirements

For Linux ASMLib utility is not required, however recommended and considered best practice. ** Not support in Red Hat 6 for ASMLib going forward based on most recent information For Linux ASMLib should be installed and configured RAW volumes presented to the host/operating System

RAW volumes partitioned

Permissions and ownership of volumes for oracle and dba or ASM Admin group Make permissions and ownership of volumes persistent across host restarts

ASMLib for Linux

Check Installation

rpm qa | grep oracleasm

http://www.oracle.com/technology/software/tech/linux/asmlib/rhel5.html

Install ASMLib

rpm -i oracleasm-support-2.1.3-1.el5.i386.rpm rpm -i oracleasm-2.6.18-128.1.6.el5-2.0.5-1.el5.i686.rpm rpm -i oracleasmlib-2.0.4-1.el5.i386.rpm rpm -i oracleasm-2.6.18-128.1.6.el5debug-2.0.5-1.el5.i686.rpm rpm -i oracleasm-2.6.18-128.1.6.el5-debuginfo-2.0.5-1.el5.i686.rpm rpm -i oracleasm-2.6.18-128.1.6.el5PAE-2.0.5-1.el5.i686.rpm rpm -i oracleasm-2.6.18-128.1.6.el5xen-2.0.5-1.el5.i686.rpm /etc/init.d/oracleasm configure

Configure ASMLib

Create disks for ASM

Partition RAW volumes to be used for ASM

fdisk /dev/sda

Change Group at min and permissions on volume and partition

chmod 660 /dev/sda chmod 660 /dev/sda1 chown oracle:dba /dev/sda chown oracle:dba /dev/sda1

Place Group and permission changes in /etc/rc.d/rc.local Initialize partitions on devices into ASM Disks

/etc/init.d/oracleasm createdisk ASMDSK1 /dev/sda1

INSTALLING ASM

Software and OS Requirements for Oracle Enterprise Linux

OS Requirements Enterprise Linux

At least 1GB of free available memory on server At least 1.8GB of free swap Oracle user Oracle dba group Ensure the following exist in the /etc/security/limits.conf
* *

hard nproc hard nofile

16384 65536

Ensure that the limits are configured to set for new sessions by ensuring the following are in the /etc/pam.d/system-auth file.
session required pam_limits.so session required pam_unix.so

January 12 / Slide 33 / Title of Document

Software and OS Requirements for Oracle Enterprise Linux

Ensure that the following are included in the .bash_profile for the oracle user
ulimit -n 65536 ulimit -u 16384

Ensure oracle primary group dba

$ usermod g dba oracle

kernel.msgmnb = 65536 kernel.msgmax = 65536 kernel.shmmax = 4294967295 kernel.shmall = 268435456 kernel.sem=250 32000 100 128 net.ipv4.ip_local_port_range = 1024 65000 < 11gR1 net.ipv4.ip_local_port_range = 9000 65500 >= 11gR2 net.core.rmem_default=4194304 net.core.wmem_default=262144 net.core.rmem_max=4194304 net.core.wmem_max=1048576 fs.file-max = 6815744 fs.aio-max-nr = 1048576
January 12 / Slide 34 / Title of Document

set the following in the /etc/sysctl.conf

Software and OS Requirements for Oracle Enterprise Linux

CD3 rpm -Uvh libaio-0 rpm -Uvh libaio-devel-0.3.105-2.i386.rpm CD4 rpm -Uvh elfutils-libelf-0 rpm -Uvh elfutils-libelf-devel-0.91.1-3.i386.rpm

CD1 rpm -Uvh unixODBC-2.2.11-1.RHEL4.1.i386.rpm

January 12 / Slide 35 / Title of Document

Installation Basics

Ensure OS compatibility for the ASM version to be installed Ensure all OS pre-reqs are satisfied

For ASM the latest ASM version is recommended to be used even for lower versions of the database software
Obtain the latest ASM Software currently 11.2 and is included in the Grid Infrastructure Software Installation The Oracle Universal Installer greatly simplifies the installation process for the Database and Grid Infrastructure software.

ASM COMMAND LINE UTILITY

What is the ASM Command Line Utility

Command line interface for management of ASM disk groups and ACFS volumes

Run from Grid Infrastructure home

<GRIDHOME>/bin/asmcmd

ASMCMD Commands

Instance Management

iostat lsct lsop

Displays statistics for mounted devices. Lists information about current ASM clients. Lists the current operations on a disk group, database, or ASM instance. lspwusr List the users from an Oracle PWFILE file. orapwusr Add, drop, or change an Oracle PWFILE user. shutdown Shuts down an ASM instance. spcopy Copies a SPFILE. startup Starts up an ASM instance.

ASMCMD Commands

File Access

chgrp chmod chown groups grpmod lsgrp lsusr mkgrp mkusr passwd rmgrp rmusr

Changes the group of a file. Changes permissions of a file. Changes the owner of a file. List the user groups that a user belongs to. Adds or removes users from an existing user group. Lists ASM access control list (ACL) groups. Lists users in a disk group. Creates a new user group. Adds a user to disk group. Changes the password of a user in a disk group. Deletes a user group. Deletes a user from a disk group.

ASMCMD Commands

File Management
cd du

find ls

lsof mkalias mkdir pwd rm rmalias

Changes the current directory to the specified directory. Displays the total disk space occupied by ASM files in the specified ASM directory and all of its subdirectories, recursively. Lists the paths of all occurrences of the specified name (with wildcards) under the specified directory. Lists the contents of an ASM directory, the attributes of the specified file, or the names and attributes of all disk groups. Lists the open files. Creates an alias for system-generated filenames. Creates ASM directories. Displays the path of the current ASM directory. Deletes the specified ASM files or directories. Deletes the specified alias, retaining the file alias points to.

ASMCMD Commands

Disk Group Management

chdg Changes a disk group (add, drop, or rebalance). chkdg Checks or repairs a disk or failure group. cp Enables you to copy files between ASM disk groups on a local instance and remote instances. dgdrop Drops a disk group. lsattr Lists the attribute and attribute settings of a disk group. lsdg Lists disk groups and their information. lsdsk Lists disks visible to ASM. lsgrp Lists ASM access control list (ACL) groups. md_backup Creates a backup of all of the mounted disk groups. md_restore Restores disk groups from a backup. mkdg Creates a disk group. mount Mounts a disk group.

ASMCMD Commands

Disk Group Management (cont.)

offline online rebal remap setattr umount

Offline a disk or a failure group that belongs to a disk group. Onlines a disk or a failure group that belongs to a disk group. Rebalances a disk group. Repairs a range of physical blocks on a disk. Sets attributes in an ASM disk group. Dismounts a disk group.

ASMCMD Commands

ACFS ASM Clustered File System

volcreate voldelete voldisable volenable volinfo volresize volset volstat

Create a volume Delete a volume Disable a volume. Enable a Volume Volume Information, Resize a volume Change existing attribute Report volume I/O statistics.

WORKING WITH ASM DISK GROUPS

January 12 / Slide 45 / Title of Document

Discovering Disks

You can discover disk groups using ASM lib or the ASM instance.

ASMLib
/etc/init.d/oracleasm scandisks /etc/init.d/oracleasm listdisks

ASM Instance

SQL> select name, path from v$asm_disk ;

Create ASM Disk Groups

Set ORACLE_HOME to Grid Infrastructure installation location

SET ORACLE_SID to ASM instance name typically +ASM

A disk group can be created using SQL Plus or the ASM command line utility.

Create ASM Disk Groups

External Redundancy

SQL Plus

SQL> create diskgroup UNDO external redundancy disk 'ORCL:UNDO1' ;

asmcmd
Create diskgroup_config.xml <dg name="dgroup1" redundancy=external"> <dsk string="/dev/raw/disk1" /> <a name="compatible.asm" value="11.2"/> <a name="compatible.rdbms" value="11.2"/> </dg> ASMCMD> mkdg diskgroup_config.xml

Create ASM Disk Groups

Normal Redundancy

SQL Plus

SQL> create diskgroup test_failgroup normal redundancy FAILGROUP fg1 DISK ORCL:ASMDSK1' NAME ASMDSK1 FAILGROUP fg2 DISK ORCL:ASMDSK2' NAME ASMDSK2 ;

asmcmd

Create diskgroup_config.xml
<dg name="dgroup1" redundancy="normal"> <fg name="fg1"> <dsk string="/dev/raw/disk1" /> <dsk string="/dev/raw/disk2" /> </fg> <fg name="fg2"> <dsk string="/dev/raw/disk3" /> <dsk string="/dev/raw/disk4" /> </fg> <a name="compatible.asm" value="11.2"/> <a name="compatible.rdbms" value="11.2"/> </dg>

ASMCMD> mkdg diskgroup_config.xml

Create ASM Disk Groups

High Redundancy

SQL Plus

SQL> create diskgroup test_failgroup high redundancy FAILGROUP fg1 DISK '/dev/oracleasm/disks/ASMDSK1' NAME ASMDSK1 FAILGROUP fg2 DISK '/dev/oracleasm/disks/ASMDSK2' NAME ASMDSK2 FAILGROUP fg3 DISK '/dev/oracleasm/disks/ASMDSK2' NAME ASMDSK3 ;

Create ASM Disk Groups

asmcmd

Create diskgroup_config.xml
<dg name="dgroup1" redundancy=high"> <fg name="fg1"> <dsk string="/dev/raw/disk1" /> <dsk string="/dev/raw/disk2" /> </fg> <fg name="fg2"> <dsk string="/dev/raw/disk3" /> <dsk string="/dev/raw/disk4" /> </fg> <fg name="fg3"> <dsk string="/dev/raw/disk5" /> <dsk string="/dev/raw/disk6" /> </fg> <a name="compatible.asm" value="11.2"/> <a name="compatible.rdbms" value="11.2"/> </dg>

ASMCMD> mkdg diskgroup_config.xml

Mounting/Un-mounting Disk Groups

Mounting and Un-mounting Disk Groups can be done via SQL Plus or the asmcmd utility

Mount

SQL Plus

SQL> alter diskgroup undo mount ; asmcmd> mount undo ;

asmcmd

Un-mount

SQL Plus

SQL> alter diskgroup undo dismount ; asmcmd> umount undo ;

asmcmd

Add Disk/Space to ASM Disk Group

Space is added to a disk group by the addition of ASM disks/volumes.

SQL Plus

SQL> alter diskgroup test add disk ORCL:ASMDSK5

Remove Disk/Space from ASM Disk Group

Space is removed from an ASM Disk Group by removing disks from the diskgroup.

SQL Plus

SQL> alter diskgroup test_failgroup drop disk ASMDSK2 ;

Change ASM Disk Group Compatibility

The Disk Group Compatibility can be change after the disk group is created provided it does not exceed compatibility in the environment being used. The COMPATIBLE.ASM and COMPATIBLE.RDBMS dictate the instance and RDBMS capability instance parameter settings.
COMPATIBLE.ASM 10.1 COMPATIBLE.RDBMS 10.1 ASM Instance >=10.1 RDBMS Instance >=10.1

11.1
11.2 11.2

10.1
11.1 11.2

>=11.1
>=11.2 >=11.2

>=10.1
>=11.1 >=11.2

Disk Group can only be changed upward (so be careful)

SQL Plus

SQL> ALTER DISKGROUP data SET ATTRIBUTE 'compatible.asm' = '11.1';

Rename Disk Group

Starting will 11g Release 2 a disk group can be renamed using the renamedg command line utility

asmcmd

renamedg dgname=olddiskgroup newdgname=newdiskgroup

** Note: If Disk Group Contains Database Files the database files will need to be relocated. This will require off lining the tablespaces that have datafiles on the diskgroup or shutting the database down prior to renaming the diskgroup. Then once diskgroup is rename relocating all files with alter database rename command for new Disk Group Name.

Managing ASM File Access Control for Disk Groups

Managing file and directory is done using the asmcmd command line utility

Managing access in asm is much like managing file and directory access in unix operating systems.
You have users, groups, directory and file permissions controlled through the use of commands

mkusr mkgrp rmgrp rmusr chown chgrp chmod

Managing ASM File Access Control for Disk Groups

mkusr Create New User

asmcmd> mkusr dg1 myuser ; asmcmd> mkgrp dg1 mygroup ; -a Add User

mkgrp Create New Group

grpmod Add/Remove Users for Group

-d Remove User

grpmod a dg1 mygroup myuser grpmod d dg1 mygroup myuser

rmgrp Remove Group

asmcmd> rmgrp dg1 mygroup ; asmcmd> rmusr dg1 myuser ;

rmusr Remove User

Managing ASM File Access Control for Disk Groups

chown

ASMCMD> chown db1 +data/orcl/datafile/* ASMCMD> chown R db1 +data/orcl ASMCMD> chown db1:backup_user +data/controlfile.f asmcmd> chgrp backup_users +data/controlfile.f ; asmcmd> chgrp R backup_users +data/* ; ASMCMD> chmod 644 +data/controlfile.f ASMCMD> chmod go+rw +data/controlfile.f ASMCMD> chmod ug-w +data/orcl/datafile/* ASMCMD> chmod R 600 +data/orcl

chgrp

chmod

Drop ASM Disk Group

A disk group can be dropped using either SQL Plus or the asmcmd command line utility

SQL Plus
DROP DISKGROUP dg1 ; ** If Disk Group has files then use the INCLUDING CONTENTS clause SQL> DROP DISKGROUP disk_group_1 INCLUDING CONTENTS;

asmcmd
asmcmd> dgdrop dg1 ; ** If Disk Group has files then use the r option asmcmd> dgdrop r dg1 ;

ASM CLUSTER FILE SYSTEM

Overview

Requires installation of 11g Release 2 (11.2) Grid Infrastructure Software ACFS executes as a UNIX (POSIX and X/OPEN compatible) file system for Linux and UNIX and ACFS works as a Windows Filesystem on Windows. Applications and OS commands work natively against ACFS Exabyte capable files and file system capacities on 64 bit platforms Peer to peer, multi-node shared filesystem with direct, coherent and cache I/O to ASM Storage. Works as a Clustered or non-clustered filesystem Oracle ACFS are provided with single node file system coherency for single host or when distributed across cluster members. Oracle ACFS file system can be used for an Oracle Database Home file system.

Oracle ACFS Driver Model

Oracle ACFS is installed as a dynamically loadable Operating System filesystem driver.

Driver is implemented as a Virtual File System (VFS)

Integrated with the native operating system

Oracle ACFS Mount Model and Namespace

Hierarchical filesystem

Files Sub-directories Files are the leaf nodes Single-filesystem naming model

Tree-structured Namespace

Not designed to be a root filesystem or bootable.

Oracle ACFS Mount Registry

Multi-platform mount registration facility Automatically mount file systems on all nodes in a cluster that are registered Clustered and non-clustered filesystems are registered in the Mount Registry, most useful for clusters really

Clustered see acfsutil command Non-clustered see srvctl add filesystem command

Creates the mount point if it does not exist Registry mount actions will automatically mount associated ASM disk groups

Oracle ACFS Snapshots

Point in time copy of an ACFS filesystem Works very similar to SAN Volume or Flash Copies

Snapshots are immediately available for use after creation

Always online while the file system is mounted Snapshot storage is maintained within the file system

Snapshots are administered with the acfsutil snap commands

63 snapshot views supported for each file system

Oracle ACFS and Backup and Restore

Uses same backup capabilities as other filesystems for the host operating system environment.

ACFS snapshots can be dynamically created and used to present a consistent view of an active file system for a backup
NOTE: Windows backup applications that depend upon reparse points or the Windows Volume Shadow Copy Service (VSS) may not be supported.

Oracle ACFS Integration with ASM

ACFS requires ASM for storage Always integrated with ASM

Configured with a traditional device file

ACFS filesystem inherits ASM storage management features like balanced distribution, mirroring, dynamic resizing, etc.

Driver established connection with ASM instance

Always unmount ACFS filesystems prior to ASM instance shutdown or prior t0 unmounting a disk group to avoid the filesystem being put into an error state.

Loading ACFS Drivers / Modules

To utilize ACFS the ACFS drivers and modules must be loaded into the operating system

For Grid Infrastructure installations in a cluster the drivers and modules are loaded automatically Single instance Grid Infrastructure installations the ACFS Drivers/Modules must be loaded Manaually
Log into the host operating system as root <GRIDHOME>/bin/acfsload start Place the load command into /etc/rc.d/rc.local for the load to be persistent across node restarts

Load ACFS Drivers/Modules Manually

Create/Manage ACFS File System

Must first have a Disk Group for the ACFS file system to use Disk Group Compatibility must be 11.2.0 or higher

SQL> select group_number, name, compatibility, database_compatibility from v$asm_diskgroup; alter diskgroup avm SET ATTRIBUTE 'compatible.advm' = '11.2.0.0' Create using asmcmd ASM command line utility

Create an ACFS File System through ASMCA or asmcmd

volcreate -d diskgroupa -s 10G volume

Get ACFS volume information

volinfo a dg1 All - volenable a dg1 Volume volenable d dg1 <volumename>

Enable volumes in a Disk Group

Create/Manage ACFS File System

Resize a ACFS volume in a Disk Group

volresize -d <diskgroupname> -s 2G [ -f ] <volumename> volstat -d <diskgroup_name> <volume_name> voldisable -d <diskgroup_name> <volume_name> voldelete -d <diskgroup_name > <volume_name>

ACFS Volume Statistics

Disable ACFS volume

Remove an ACFS volume

Thank You

Thank you!

January 12 / Slide 72 / Title of Document