How ASM disk header block repair works

================================
In this post, I will explain one of my last works about the ASM disk header bloc
k.
First, I will create a TEST tablespace in my orcl database. The TEST s datafile wi
ll be managed by ASM.
[oracle@oel ~]$ . oraenv
ORACLE_SID = [+ASM] ? orcl
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@oel ~]$ sqlplus / as sysdba
SQL> create tablespace test datafile '+data' size 5M autoextend on maxsize unlim
ited;
Tablespace created.
SQL> select instance_name from v$instance;
INSTANCE_NAME
---------------orcl
SQL> select file_id,tablespace_name,file_name,status,online_status from dba_data
_files
2 /
FILE_ID TABLESPACE_NAME
STATUS
ONLINE_
---------- ----------------------------------------- --------- ------4 USERS
AVAILABLE ONLINE
3 UNDOTBS1
AVAILABLE ONLINE
2 SYSAUX
AVAILABLE ONLINE
1 SYSTEM
AVAILABLE SYSTEM
5 EXAMPLE
AVAILABLE ONLINE
6 TEST
AVAILABLE ONLINE

FILE_NAME
-------------------------------------/u02/oradata/orcl/users01.dbf
/u02/oradata/orcl/undotbs01.dbf
/u02/oradata/orcl/sysaux01.dbf
/u02/oradata/orcl/system01.dbf
/u02/oradata/orcl/example01.dbf
+DATA/orcl/datafile/test.258.798578613

6 rows selected.
My ASM instance manages two diskgroups DATA (external redundancy) and RL (normal
redundancy).
DATA is a 2 disks diskgroup managed by asmlib.
SQL> select group_number,name,type,state from v$asm_diskgroup
2 /
GROUP_NUMBER NAME TYPE STATE
------------ ---- ------ ----------1 DATA EXTERN MOUNTED
2 RL NORMAL MOUNTED
SQL> select path from v$asm_disk where group_number=1;

PATH
--------------------------------ORCL:ASM1
ORCL:ASM2
And now, we erase the header block of each disk. Header block is the first one o
f an ASM disk device, and its default size is 4096 bytes (This size is available
in _asm_blksize undocumented parameter).
[oracle@oel ~]$
1+0 records in
1+0 records out
4096 bytes (4.1
[oracle@oel ~]$
1+0 records in
1+0 records out
4096 bytes (4.1

dd if=/dev/zero of=/dev/oracleasm/disks/ASM1 bs=4096 count=1

kB) copied, 2.2e-05 seconds, 186 MB/s
dd if=/dev/zero of=/dev/oracleasm/disks/ASM2 bs=4096 count=1
kB) copied, 0.000336 seconds, 12.2 MB/s

Even my block headers destroyed, I can still write in my diskgroup and allocate
extents in it.
SQL> alter database datafile 6 resize 10M;
Database altered.
SQL> create table t tablespace TEST as select * from dba_source;
Table created.
SQL> alter system checkpoint;
System altered.
SQL> select file#,checkpoint_time from v$datafile_header where file#=6;
FILE# CHECKPOINT_TIME
---------- ------------------6 05/11/2012 19:27:43
Well, let s try to restart the rdbms instance:
SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup
ORACLE instance started.
Total System Global Area 1068937216
Fixed Size
2235208
Variable Size
377488568
Database Buffers
683671552
Redo Buffers
5541888
Database mounted.
Database opened.
SQL> select count(*) from t;
COUNT(*)
---------702070

bytes
bytes
bytes
bytes
bytes

Damned ! I can still restart it and read all file extents.

There s something strange in this demo, my ASM disk header is invalid and I still
can read files. I will verify with kfed my headers state (only one ASM disk is s
hown below):
oracle@oel ~]$ kfed read /dev/oracleasm/disks/ASM1
kfbh.endian:
0 ; 0x000: 0x00
kfbh.hard:
0 ; 0x001: 0x00
kfbh.type:
0 ; 0x002: KFBTYP_INVALID
kfbh.datfmt:
0 ; 0x003: 0x00
kfbh.block.blk:
0 ; 0x004: blk=0
kfbh.block.obj:
0 ; 0x008: file=0
kfbh.check:
0 ; 0x00c: 0x00000000
kfbh.fcn.base:
0 ; 0x010: 0x00000000
kfbh.fcn.wrap:
0 ; 0x014: 0x00000000
kfbh.spare1:
0 ; 0x018: 0x00000000
kfbh.spare2:
0 ; 0x01c: 0x00000000
2B469E002400 00000000 00000000 00000000 00000000 [................]
Repeat 255 times
KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlo
ck][Invalid OSM block type][][0]
[oracle@oel ~]$ sudo /u01/app/oracle/product/11.2.0.3/dbhome_1/bin/kfed read /de
v/sdb1
kfbh.endian:
0 ; 0x000: 0x00
kfbh.hard:
0 ; 0x001: 0x00
kfbh.type:
0 ; 0x002: KFBTYP_INVALID
kfbh.datfmt:
0 ; 0x003: 0x00
kfbh.block.blk:
0 ; 0x004: blk=0
kfbh.block.obj:
0 ; 0x008: file=0
kfbh.check:
0 ; 0x00c: 0x00000000
kfbh.fcn.base:
0 ; 0x010: 0x00000000
kfbh.fcn.wrap:
0 ; 0x014: 0x00000000
kfbh.spare1:
0 ; 0x018: 0x00000000
kfbh.spare2:
0 ; 0x01c: 0x00000000
2B413559C400 00000000 00000000 00000000 00000000 [................]
Repeat 255 times
KFED-00322: file not found; arguments: [kfbtTraverseBlock] [Invalid OSM block ty
pe] [] [0]
So KFED has confirmed my header blocks are invalid. Now, I will try to unmount a
nd remount the diskgroup to see if there s any effect on diskgroup mount operation
.
[oracle@oel ~]$ sqlplus / as sysasm
SQL> alter diskgroup DATA dismount;
Diskgroup altered.
SQL> alter diskgroup DATA mount;
alter diskgroup DATA mount
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15017: diskgroup "DATA" cannot be mounted
ORA-15063: ASM discovered an insufficient number of disks for diskgroup "DATA"

So ASM disk header seems to be important for mounting the diskgroup but without
effect on asm extent allocation etc.
Now I will repair asm disks with kfed and repair operation to restore my asm hea
der blocks:
[oracle@oel ~]$ kfed repair /dev/oracleasm/disks/ASM1
[oracle@oel ~]$ kfed repair /dev/oracleasm/disks/ASM2
[oracle@oel ~]$ sqlplus / as sysasm
SQL> alter diskgroup DATA mount;
Diskgroup altered.
So KFED is able to restore the header (with the repair operation).
[oracle@oel ~]$ . oraenv
ORACLE_SID = [+ASM] ? orcl
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@oel ~]$ sqlplus / as sysdba
Connected to an idle instance.
SQL> startup
...
Database opened.
SQL> select count(*) from T;
COUNT(*)
---------702070
And datas in the T table are still there

no problem !

Well, let s summarize

I erased the header block of both asm disks in my diskgroup
without any impact on file extent allocation. The only impact I had was on the m
ount capability. Next, I have restored the header block of each disk with a kfed
repair operation yes, but without any backup of the disk or the disk header. So
I wonder where was the backup of my header block?
First, I will try a very simple method. I will have a look at all metadata block
s in my asm disk device. I hope I will find another header block. For this purpo
se, i will use a basic shell script to analyze each block. If the script finds a
block with the type KFBTYP_DISKHEAD, it will keep the block position, and print
them at the end of script execution:
The basic script:
#!/bin/bash
#set -x
export ORAENV_ASK=NO
export ORACLE_SID=+ASM
. oraenv
i=0
ret=0
tab_cnt=0;
blk_typ='';

while [ "$blk_typ" != "KFBTYP_INVALID" ];

do
blk_typ=`kfed read $1 blkn=$i | grep kfbh.type | awk '{print $5;}' | sed 's/^
*//g' | sed 's/ *$//g' `
ret=$?
if [ "$blk_typ" = "KFBTYP_DISKHEAD" ]; then
t[$i]=$i
fi
let i=$i+1
done
echo "list of header block with KFBTYP_DISKHEAD type"
echo ${t[@]}
Execution results:
[oracle@oel ~]$ ./asm_surgery.sh /dev/oracleasm/disks/ASM1
The Oracle base remains unchanged with value /u01/app/oracle
list of header block with KFBTYP_DISKHEAD type
0 510
[oracle@oel ~]$ ./asm_surgery.sh /dev/oracleasm/disks/ASM2
The Oracle base remains unchanged with value /u01/app/oracle
list of header block with KFBTYP_DISKHEAD type
0 510
Ok, so there is a copy of the header block in the 510th blocks in each disk.
To double check this, I will erase the header block of my first ASM disk, and us
e kfed repair and strace to see what really happens:
[oracle@oel ~]$ dd if=/dev/zero of=/dev/oracleasm/disks/ASM1 bs=4096 count=1
1+0 records in
1+0 records out
4096 bytes (4.1 kB) copied, 2.3e-05 seconds, 178 MB/s
[oracle@oel ~]$ strace kfed repair /dev/oracleasm/disks/ASM1
.../...
stat("/dev/oracleasm/disks/ASM1", {st_mode=S_IFBLK|0660, st_rdev=makedev(8, 17),
...}) = 0
access("/dev/oracleasm/disks/ASM1", F_OK) = 0
statfs("/dev/oracleasm/disks/ASM1", {f_type=0x958459f6, f_bsize=4096, f_blocks=0
, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0, f_fsid={0, 0}, f_namelen=255, f_f
rsize=4096}) = 0
open("/dev/oracleasm/disks/ASM1", O_RDWR) = 7
lseek(7, 2088960, SEEK_SET)
= 2088960
read(7, "\1\202\1\1\376\200<\206\371\7"..., 4096) = 4096
lseek(7, 0, SEEK_SET)
= 0
read(7, ""..., 4096) = 4096
lseek(7, 0, SEEK_SET)
= 0
write(7, "\1\202\1\1\200\302\206\371\7"..., 4096) = 4096
close(7)
= 0
This is very interesting.
First operation, it opens the device and read the block after 2088960 bytes. But
2088960/4096 = 510, so it reads the 510th block of the disk. Next it reads the
block at position 0 (header block), and then writes in it the content of the 510
th block.

Well, now I know there s a copy of each header block in a secret position !!! (blo
ck #510) and kfed uses this block to repair the main header block.
During my tests, I noticed that sometimes, asm disks lost their asm label (I don t
know why). As a consequence, repaired disks won t be recreated by the oracleasm s
candisks (or after a reboot).
[root@oel ~]# ls /dev/oracleasm/disks/*
/dev/oracleasm/disks/ASM1 /dev/oracleasm/disks/ASM2 /dev/oracleasm/disks/ASM3
/dev/oracleasm/disks/ASM4
[root@oel ~]# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Cleaning disk "ASM1"
Scanning system for ASM disks...
[root@oel ~]# ls /dev/oracleasm/disks/*
/dev/oracleasm/disks/ASM2 /dev/oracleasm/disks/ASM3 /dev/oracleasm/disks/ASM4
According the fact that you know which device is corresponding to asmlib disk de
vice, you can relabel this disk with oracleasm renamedisk command (be very caref
ul with this command):
[root@oel ~]# oracleasm renamedisk -f /dev/sdb1 ASM1
Writing disk header: done
Instantiating disk "ASM1": done
[root@oel ~]# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
[root@oel ~]# ls /dev/oracleasm/disks/*
/dev/oracleasm/disks/ASM1 /dev/oracleasm/disks/ASM2 /dev/oracleasm/disks/ASM3
/dev/oracleasm/disks/ASM4
Comments : Leave a Comment
Categories : Oracle, Internals, ASM / ADVM, 11g
Dump ASM disk header method #2
21 03 2012
In a previous post, I explained how to dump ASM Disk header and how to analyze i
t. This method, which used kfed binary, is a little bit complex because of field
s with a complex name extracted from the internal structure.
Another way to proceed and to see main information about asm disks is to use an
undocumented binary located in $ORACLE_HOME/bin named amdu.
If amdu is not available in your $ORACLE_HOME/bin directory, you can build it wi
th the next make command:
[oracle@oel ~]$ make -f $ORACLE_HOME/rdbms/lib/ins_rdbms.mk iamdu
To dump your ASM diskheader, you just have to execute this command with the -dis
kstring option and specify the location of your ASM disk devices (default uses A
SM lib directory):
[oracle@oel ~]$ amdu -diskstring='/dev/oracleasm/disks/*'
amdu_2012_03_21_11_14_36/

Now, change directory to the output generated and have a look to the report.txt
file:
[oracle@oel ~]$ cd amdu_2012_03_21_11_14_36/
[oracle@oel amdu_2012_03_21_11_14_36]$ cat report.txt
-*-amdu-******************************** AMDU Settings ********************************
ORACLE_HOME = /u01/app/oracle/product/11.2.0.3/dbhome_1
System name: Linux
Node name: oel
Release: 2.6.18-274.18.1.0.1.el5
Version: #1 SMP Thu Feb 9 19:07:16 EST 2012
Machine: x86_64
amdu run: 21-MAR-12 11:14:36
Endianess: 1
--------------------------------- Operations --------------------------------------------------------------- Disk Selection -------------------------------diskstring '/dev/oracleasm/disks/*'
------------------------------ Reading Control ------------------------------------------------------------- Output Control ------------------------------********************************* DISCOVERY **********************************
----------------------------- DISK REPORT N0001 -----------------------------Disk Path: /dev/oracleasm/disks/ASM1
Unique Disk ID:
Disk Label:
Physical Sector Size: 512 bytes
Disk Size: 1019 megabytes
Group Name: DATA
Disk Name: DATA_0000
Failure Group Name: DATA_0000
Disk Number: 0
Header Status: 3
Disk Creation Time: 2011/05/27 10:57:54.822000
Last Mount Time: 2012/03/21 08:54:10.221000
Compatibility Version: 0x0b200000(11020000)
Disk Sector Size: 512 bytes
Disk size in AUs: 1019 AUs
Group Redundancy: 1
Metadata Block Size: 4096 bytes
AU Size: 1048576 bytes
Stride: 113792 AUs
Group Creation Time: 2011/05/27 10:57:54.456000
File 1 Block 1 location: AU 2
OCR Present: NO
----------------------------- DISK REPORT N0002 -----------------------------Disk Path: /dev/oracleasm/disks/ASM2
Unique Disk ID:
Disk Label:
Physical Sector Size: 512 bytes
Disk Size: 1019 megabytes
Group Name: DATA
Disk Name: DATA_0001
Failure Group Name: DATA_0001
Disk Number: 1
Header Status: 3
Disk Creation Time: 2011/05/27 10:57:54.822000
Last Mount Time: 2012/03/21 08:54:10.221000
Compatibility Version: 0x0b200000(11020000)
Disk Sector Size: 512 bytes

Disk size in AUs: 1019 AUs

Group Redundancy: 1
Metadata Block Size: 4096 bytes
AU Size: 1048576 bytes
Stride: 113792 AUs
Group Creation Time: 2011/05/27 10:57:54.456000
File 1 Block 1 location: AU 0
OCR Present: NO
******************************* END OF REPORT ********************************
As mentioned, now you know if your disk is part of a disk group, the DG Allocati
on Unit size, disk size, last mounted time etc.
Of course, this tool can be used regarless of the asm instance status.

Comments : 1 Comment
Categories : ASM / ADVM, Internals, Oracle
Dump ASM disk header
2 11 2011
If you want to dump ASM disk header, you can use an Oracle internal tool to obta
in information about your disk, diskgroup etc. even if the disk is offline.
This tool is named KFED (Kernel File EDitor). It is fitted by default with an Or
acle 11g installation, but you ll need to build it if you want to use it with Ora
cle 10g :
[oracle@oel ~]$ make -f $ORACLE_HOME/rdbms/lib/ins_rdbms.mk ikfed
Well, now have a closer look to a feature of this tool.
If you want to read information stored on the ASM Disk header, you can use it li
ke this :
[oracle@oel ~]$ kfed read /dev/oracleasm/disks/ASM3 dsk1.dump
Now, you have in the dsk1.dump file the content of your ASM file header
leasm/disks/ASM3?. This file can be easily read by a text editor
kfbh.endian:
1 ; 0x000: 0x01
kfbh.hard:
130 ; 0x001: 0x82
kfbh.type:
1 ; 0x002: KFBTYP_DISKHEAD
kfbh.datfmt:
1 ; 0x003: 0x01
kfbh.block.blk:
0 ; 0x004: T=0 NUMB=0x0
kfbh.block.obj:
2147483648 ; 0x008: TYPE=0x8 NUMB=0x0
kfbh.check:
2930000864 ; 0x00c: 0xaea443e0
kfbh.fcn.base:
0 ; 0x010: 0x00000000
kfbh.fcn.wrap:
0 ; 0x014: 0x00000000
kfbh.spare1:
0 ; 0x018: 0x00000000
kfbh.spare2:
0 ; 0x01c: 0x00000000
kfdhdb.driver.provstr:
ORCLDISKASM3 ; 0x000: length=12
kfdhdb.driver.reserved[0]:
860705601 ; 0x008: 0x334d5341
kfdhdb.driver.reserved[1]:
0 ; 0x00c: 0x00000000
kfdhdb.driver.reserved[2]:
0 ; 0x010: 0x00000000

/dev/orac

kfdhdb.driver.reserved[3]:
0
kfdhdb.driver.reserved[4]:
0
kfdhdb.driver.reserved[5]:
0
kfdhdb.compat:
186646528
kfdhdb.dsknum:
0
kfdhdb.grptyp:
2
kfdhdb.hdrsts:
3
kfdhdb.dskname:
MIRROR_DG_0000
kfdhdb.grpname:
MIRROR_DG
kfdhdb.fgname:
MIRROR_DG_0000
kfdhdb.capname:
kfdhdb.crestmp.hi:
32959021
R=0x7db
kfdhdb.crestmp.lo:
3500063744
NS=0x34
kfdhdb.mntstmp.hi:
32959382
AR=0x7db
kfdhdb.mntstmp.lo:
505578496
NS=0x7
kfdhdb.secsize:
512
kfdhdb.blksize:
4096
kfdhdb.ausize:
1048576
kfdhdb.mfact:
113792
kfdhdb.dsksize:
1019
kfdhdb.pmcnt:
2
kfdhdb.fstlocn:
1
kfdhdb.altlocn:
2
kfdhdb.f1b1locn:
2
kfdhdb.redomirrors[0]:
0
kfdhdb.redomirrors[1]:
0
kfdhdb.redomirrors[2]:
0
kfdhdb.redomirrors[3]:
0

;
;
;
;
;
;
;
;
;
;
;
;

0x014:
0x018:
0x01c:
0x020:
0x024:
0x026:
0x027:
0x028:
0x048:
0x068:
0x088:
0x0a8:

0x00000000
0x00000000
0x00000000
0x0b200000
0x0000
KFDGTP_NORMAL
KFDHDR_MEMBER
length=14
length=9
length=14
length=0
HOUR=0xd DAYS=0x11 MNTH=0xa YEA

; 0x0ac: USEC=0x0 MSEC=0x3af SECS=0x9 MI

; 0x0b0: HOUR=0x16 DAYS=0x1c MNTH=0xa YE
; 0x0b4: USEC=0x0 MSEC=0xa1 SECS=0x22 MI
;
;
;
;
;
;
;
;
;
;
;
;
;

0x0b8:
0x0ba:
0x0bc:
0x0c0:
0x0c4:
0x0c8:
0x0cc:
0x0d0:
0x0d4:
0x0d8:
0x0da:
0x0dc:
0x0de:

0x0200
0x1000
0x00100000
0x0001bc80
0x000003fb
0x00000002
0x00000001
0x00000002
0x00000002
0x0000
0x0000
0x0000
0x0000

Now, we can read some information about the file : on the structure kfdhdb , at the
offset 0048, and coded on 9 bytes, the name of the diskgroup which owns this ASM
disk file.
Most important information are detailed below :
* kfbh.endian: Endian used on this disk : 1 for little endian.
* kfdhdb.driver.provstr: Provision String used for ASM (which means in our case
: ORCL:DISKASM3)
* kfdhdb.grptyp: type of diskgroup the disk is attached to.
* kfdhdb.hdrsts: header status. Here, the disk is a member of the diskgroup.
* kfdhdb.dskname: disk name in the disk group
* kfdhdb.grpname: disk group name
* kfdhdb.fgname: failure group name which owns the disk
* kfdhdb.secsize: sector size
* kfdhdb.blksize: block size
* kfdhdb.ausize: allocation unit size

If you want to rename the diskgroup the disk belongs to, you can edit the dumpfi
le and use the merge command of KFED to apply changes to the disk header.
[oracle@oel ~]$ kfed merge /dev/oracleasm/disks/ASM3 text=dsk1.dump
Be careful when you use the merge command because, it seems the diskgroup name, or
disk name is coded with a fixed length, so if you change the name, and this one
is based on a 4 bytes word, rename it to a 4 bytes word.
Off course, using kfed is not supported by Oracle.
Comments : Leave a Comment
Categories : ASM / ADVM, Internals, Oracle
Create or modify your virtualBox disks to be used in a RAC Cluster
18 05 2011
If you want to create a disk to be used in a RAC Cluster, you have to flag your
disk to be shared by many virtual machines.
Currently, it is not possible to create a shared disk by using the GUI.So, you m
ust use the VirtualBox command line tool : VBoxManage.
Your disk has be a fixed size disk, and you have to create it first and modify i
t next to make it shareable.
To create a shared disk, use the createhd command :
VBoxManage createhd --filename /VDisks/asmdisk1.vdi --size 5120 --format VDI --v
ariant Fixed
Next, use the option type of the modifyhd command. You can use this command to t
ransform a disk created with the GUI to be shareable.
VBoxManage modifyhd /VDisks/asmdisk1.vdi --type shareable
Now, you can attach your disk to multiple virtual machines and use it as a share
d storage device.
Comments : Leave a Comment
Categories : ASM / ADVM, RAC, VirtualBox
ORA-01031: Insufficient Privileges while creating a 11.2.0.2 Database with ASM
4 05 2011
I recently encountered this strange error during a database creation process on
my RAC :
How to resolve Could not validate ASMSNMP password due to following error- "ORA01031: insufficient privileges"
I engaged the search mode, and I found a message in my ASM instance alert.log :
ORA-01918: user 'ASMSNMP' does not exist
By a mysterious way, the ASMSNMP user hasn t been created during the Grid Infrastr
ucture installation process.
So, to resolve this problem, I have executed the steps above to create the ASMS
NMP user:

1- Create or replace the existing password file for the ASM instance
[grid@rac1 ~]$ orapwd file=/u01/app/11.2.0/grid/dbs/orapw+ASM password=oracle
2- Copy this password file on every node of your cluster
[grid@rac1 dbs]$ scp orapw+ASM grid@rac2:/u01/app/11.2.0/grid/dbs
orapw+ASM
100% 1536
1.5KB/s

00:00

3- Create the ASMSNMP user in the ASM instance, and grant it the SYSDBA Privileg
e.
[grid@rac1 dbs]$ sqlplus / as sysasm
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
SQL> create user asmsnmp identified by oracle ;
User created.
SQL> grant sysdba to asmsnmp;
Grant succeeded.
Don t forget
the ASM instance parameter REMOTE_LOGIN_PASSWORDFILE
XCLUSIVE or you will raise an ORA-01999 error.

has to be set to E

When the ASMSNMP user has been created, I have been able to create my clustered
database and to connect to ASM instance.