Академический Документы
Профессиональный Документы
Культура Документы
Contents 1 Introduction................................................................................................3 2 Initial Requests...........................................................................................3 3 Tape Backup Procedure............................................................................3 4 Disk Mirroring.............................................................................................5 5 References................................................................................................12
1
1.1
Introduction
This document describes the procedure we used for recreating HDD mirroring on relay server nodes that are part of Ericsson EMO system. Procedure shows how to perform a tape backup of root partition, and how to set up a two disk mirror of root and swap slices.
Initial Requests
Two out of three EMO relay server nodes needed to have disk mirroring restored. Relay servers are set up using Sun Fire V210 servers, with Solaris 9 being used as an OS. Each of the servers has two internal 72GB SCSI HDDs. Original set up was performed as to include mirroring of both root and swap partitions. First disk data was in order on all three relay servers, and the nodes were operational all the time. Initial requests were:
Back up root partition data on all three relay servers, before performing any disk mirroring operation. Restore disk mirroring on two relay servers, as to reflect original mirroring set up. No service downtime was a prerequisite, because the operations were performed out of Maintenance window. Since at least two relay nodes were operational at each moment, it was not difficult to meet this request.
After performing devfsadm, the content of /dev/rmt directory should look something like this:
root@rel3 # ls -l /dev/rmt/
total 48 lrwxrwxrwx 1 root other 44 Jan 15 11:13 0 -> ../../devices/pci@1c,600000/scsi@2,1/st@5,0: lrwxrwxrwx 1 root other 45 Jan 15 11:13 0b -> ../../devices/pci@1c,600000/scsi@2,1/st@5,0:b lrwxrwxrwx 1 root other 46 Jan 15 11:13 0bn -> ../../devices/pci@1c,600000/scsi@2,1/st@5,0:bn lrwxrwxrwx 1 root other 45 Jan 15 11:13 0c -> ../../devices/pci@1c,600000/scsi@2,1/st@5,0:c lrwxrwxrwx 1 root other 46 Jan 15 11:13 0cb -> ../../devices/pci@1c,600000/scsi@2,1/st@5,0:cb lrwxrwxrwx 1 root other 47 Jan 15 11:13 0cbn -> ../../devices/pci@1c,600000/scsi@2,1/st@5,0:cbn lrwxrwxrwx 1 root other 46 Jan 15 11:13 0cn -> ../../devices/pci@1c,600000/scsi@2,1/st@5,0:cn lrwxrwxrwx 1 root other 45 Jan 15 11:13 0h -> ../../devices/pci@1c,600000/scsi@2,1/st@5,0:h lrwxrwxrwx 1 root other 46 Jan 15 11:13 0hb -> ../../devices/pci@1c,600000/scsi@2,1/st@5,0:hb lrwxrwxrwx 1 root other 47 Jan 15 11:13 0hbn -> ../../devices/pci@1c,600000/scsi@2,1/st@5,0:hbn lrwxrwxrwx 1 root other 46 Jan 15 11:13 0hn -> ../../devices/pci@1c,600000/scsi@2,1/st@5,0:hn lrwxrwxrwx 1 root other 45 Jan 15 11:13 0l -> ../../devices/pci@1c,600000/scsi@2,1/st@5,0:l lrwxrwxrwx 1 root other 46 Jan 15 11:13 0lb -> ../../devices/pci@1c,600000/scsi@2,1/st@5,0:lb lrwxrwxrwx 1 root other 47 Jan 15 11:13 0lbn -> ../../devices/pci@1c,600000/scsi@2,1/st@5,0:lbn lrwxrwxrwx 1 root other 46 Jan 15 11:13 0ln -> ../../devices/pci@1c,600000/scsi@2,1/st@5,0:ln lrwxrwxrwx 1 root other 45 Jan 15 11:13 0m -> ../../devices/pci@1c,600000/scsi@2,1/st@5,0:m lrwxrwxrwx 1 root other 46 Jan 15 11:13 0mb -> ../../devices/pci@1c,600000/scsi@2,1/st@5,0:mb lrwxrwxrwx 1 root other 47 Jan 15 11:13 0mbn -> ../../devices/pci@1c,600000/scsi@2,1/st@5,0:mbn lrwxrwxrwx 1 root other 46 Jan 15 11:13 0mn -> ../../devices/pci@1c,600000/scsi@2,1/st@5,0:mn lrwxrwxrwx 1 root other 45 Jan 15 11:13 0n -> ../../devices/pci@1c,600000/scsi@2,1/st@5,0:n lrwxrwxrwx 1 root other 45 Jan 15 11:13 0u -> ../../devices/pci@1c,600000/scsi@2,1/st@5,0:u lrwxrwxrwx 1 root other 46 Jan 15 11:13 0ub -> ../../devices/pci@1c,600000/scsi@2,1/st@5,0:ub lrwxrwxrwx 1 root other 47 Jan 15 11:13 0ubn -> ../../devices/pci@1c,600000/scsi@2,1/st@5,0:ubn lrwxrwxrwx 1 root other 46 Jan 15 11:13 0un -> ../../devices/pci@1c,600000/scsi@2,1/st@5,0:un
/dev/rmt ls output can be used to identify the device name of the tape drive. The default tape drive is the /dev/rmt/0 device. 02. Feed a tape to the tape drive, and check the tape drive status.
root@rel3 # mt status Quantum Super DLT 320 tape drive: sense key(0x0)= No Additional Sense residual= 0 retries= 0 file no= 0 block no= 0
In our case root and swap file systems were mounted on c0t0d0 disk. This can be checked by using /etc/vfstab file.
root@rel3 # cat /etc/vfstab #device device mount FS fsck mount mount #to mount to fsck point type pass at boot options # fd /dev/fd fd no /proc /proc proc no /dev/dsk/c0t0d0s1 swap no -
ufs
no
When performing a full root file system backup, issue the following command:
root@rel3 # ufsdump 0ucf /dev/rmt/0 / DUMP: DUMP: DUMP: DUMP: DUMP: DUMP: DUMP: DUMP: DUMP: DUMP: Date of this level 0 dump: Fri 15 Jan 2010 12:35:27 PM CET Date of last level 0 dump: the epoch Dumping /dev/rdsk/c0t0d0s0 (rel3:/) to /dev/rmt/0. Mapping (Pass I) [regular files] Mapping (Pass II) [directories] Writing 63 Kilobyte records Estimated 19736342 blocks (9636.89MB). Dumping (Pass III) [directories] Dumping (Pass IV) [regular files] Tape rewinding
DUMP: 19736260 blocks (9636.85MB) on 1 volume at 17889 KB/sec DUMP: DUMP IS DONE DUMP: Level 0 dump on Fri 15 Jan 2010 12:35:27 PM CET
- 0 specifies that this is a 0 level dump (or a full backup) - u specifies that the /etc/dumpdates file is updated with the date of this backup - c identifies a cartridge tape device - f /dev/rmt/0 identifies the tape device - / is the file system being backed up 07. To verify that the backup was successful, use ufsrestore.
root@rel3 # ufsrestore tf /dev/rmt/0 2 . 3 ./lost+found 6016 ./var 6017 ./var/sadm 6018 ./var/sadm/install 6019 ./var/sadm/install/admin
4
1.5
Disk Mirroring
Since one of initial requests was to perform disk mirroring as to reflect original mirroring set up, here are outputs of metadb i and metastat commands, and of /etc/vfstab file that show the original setup.
root@rel2 # metadb -i flags first blk block count
a m p luo 16 8192 /dev/dsk/c0t0d0s7 a p luo 8208 8192 /dev/dsk/c0t0d0s7 a p luo 16400 8192 /dev/dsk/c0t0d0s7 a p luo 16 8192 /dev/dsk/c0t1d0s7 a p luo 8208 8192 /dev/dsk/c0t1d0s7 a p luo 16400 8192 /dev/dsk/c0t1d0s7 r - replica does not have device relocation information o - replica active prior to last mddb configuration change u - replica is up to date l - locator for this replica was read successfully c - replica's location was in /etc/lvm/mddb.cf p - replica's location was patched in kernel m - replica is master, this is replica selected as input W - replica has device write errors a - replica is active, commits are occurring to this replica M - replica had problem with master blocks D - replica had problem with data blocks F - replica had format problems S - replica is too small to hold current data base R - replica had device read errors
metadb output shows that slices 7 (/dev/dsk/cxt0d0s7) are used for storing metastat data bases, and that each slice contains three copies of meta DB.
root@rel2 # metastat d10: Mirror Submirror 0: d11 State: Okay Submirror 1: d12 State: Okay Pass: 1 Read option: roundrobin (default) Write option: parallel (default) Size: 16780224 blocks (8.0 GB) d11: Submirror of d10 State: Okay Size: 16780224 blocks (8.0 GB) Stripe 0: Device Start Block Dbase State Reloc Hot Spare c0t0d0s1 0 No Okay Yes d12: Submirror of d10 State: Okay Size: 16780224 blocks (8.0 GB) Stripe 0: Device Start Block Dbase State Reloc Hot Spare c0t1d0s1 0 No Okay Yes d0: Mirror Submirror 0: d1 State: Okay Submirror 1: d2 State: Okay Pass: 1 Read option: roundrobin (default) Write option: parallel (default) Size: 125510784 blocks (59 GB) d1: Submirror of d0 State: Okay Size: 125510784 blocks (59 GB) Stripe 0: Device Start Block Dbase State Reloc Hot Spare c0t0d0s0 0 No Okay Yes d2: Submirror of d0 State: Okay Size: 125510784 blocks (59 GB) Stripe 0: Device Start Block Dbase State Reloc Hot Spare c0t1d0s0 0 No Okay Yes Device Relocation Information: Device Reloc Device ID c0t1d0 Yes id1,sd@w4849544143484920485553313033303733464c3338303020563357595a394841 c0t0d0 Yes id1,sd@w4849544143484920485553313033303733464c33383030205633575956485641
root@rel2 # cat /etc/vfstab #device device mount FS fsck mount mount #to mount to fsck point type pass at boot options # fd /dev/fd fd no /proc /proc proc no #/dev/dsk/c0t0d0s1 swap no /dev/md/dsk/d0 /dev/md/rdsk/d0 / ufs 1 no /dev/md/dsk/d10 swap no swap /tmp tmpfs yes -
d0 mirror is used for / (root) file system, with d1 being c0t0d0 based submirror and d2 being c0t1d0 based submirror of d0. d10 is used for swap file system, with d11 being c0t0d0 based submirror and d12 being c0t1d0 based submirror of d10. Though our servers run Solaris 9 operating system, one should be able to use the following procedure on servers running Solaris 8 and Solaris 10 as well. 1.6 Root partition mirroring 01. Before you actually start performing hard disk mirroring, create backup copies of /etc/vfstab and /etc/system files:
cp -p /etc/system /etc/system.orig cp -p /etc/vfstab /etc/vfstab.orig
This can show as a useful thing if something goes wrong with /etc/system file during the process of mirroring. By issuing boot -a from the OK prompt and specifying /etc/system.orig as a system file that should be used, you should be able to boot Solaris at any moment. 02. Check slices structure (partition table) on c0t0d0 (boot) disk.
root@rel3 # format Searching for disks...done AVAILABLE DISK SELECTIONS: 0. c0t0d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424> /pci@1c,600000/scsi@2/sd@0,0 1. c0t1d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424> /pci@1c,600000/scsi@2/sd@1,0 Specify disk (enter its number): 0 selecting c0t0d0 [disk formatted] Warning: Current Disk has mounted partitions. FORMAT MENU: disk - select a disk type - select (define) a disk type partition - select (define) a partition table current - describe the current disk format - format and analyze the disk repair - repair a defective sector label - write label to the disk analyze - surface analysis defect - defect list management backup - search for backup labels verify - read and display labels save - save new disk/partition definitions inquiry - show vendor, product and revision volname - set 8-character volume name !<cmd> - execute <cmd>, then return quit format> ver Primary label contents: Volume name = < >
ascii name = <SUN72G cyl 14087 alt 2 hd 24 sec 424> pcyl = 14089 ncyl = 14087 acyl = 2 nhead = 24 nsect = 424 Part Tag Flag Cylinders Size Blocks 0 root wm 1649 - 13982 59.85GB (12334/0/0) 125510784 1 swap wu 0 - 1648 8.00GB (1649/0/0) 16780224 2 backup wm 0 - 14086 68.35GB (14087/0/0) 143349312 3 unassigned wm 13983 - 14034 258.38MB (52/0/0) 529152 4 unassigned wm 0 0 (0/0/0) 0 5 unassigned wm 0 0 (0/0/0) 0 6 unassigned wm 0 0 (0/0/0) 0 7 unassigned wm 14035 - 14086 258.38MB (52/0/0) 529152
03. Transfer the partition table from drive c0t0d0 to drive c0t1d0.
root@rel3 # prtvtoc /dev/rdsk/c0t0d0s2 | fmthard -s - /dev/rdsk/c0t1d0s2 fmthard: New volume table of contents now in place.
VTOC is volume table of contents. 04. Next, metastat data bases should be created.
root@rel3 # metadb -a -c 3 -f c0t0d0s7 c0t1d0s7 mdmonitord daemon pid 20005 already running
The -c 3 creates three copies of the metastat data base in space specified after f i.e. in slices c0t0d0s7 and c0t1d0s7. 05. Check metastat data bases status.
root@rel3 # metadb -i flags first blk block count a u 16 8192 /dev/dsk/c0t0d0s7 a u 8208 8192 /dev/dsk/c0t0d0s7 a u 16400 8192 /dev/dsk/c0t0d0s7 a u 16 8192 /dev/dsk/c0t1d0s7 a u 8208 8192 /dev/dsk/c0t1d0s7 a u 16400 8192 /dev/dsk/c0t1d0s7 r - replica does not have device relocation information o - replica active prior to last mddb configuration change u - replica is up to date l - locator for this replica was read successfully c - replica's location was in /etc/lvm/mddb.cf p - replica's location was patched in kernel m - replica is master, this is replica selected as input W - replica has device write errors a - replica is active, commits are occurring to this replica M - replica had problem with master blocks D - replica had problem with data blocks F - replica had format problems S - replica is too small to hold current data base R - replica had device read errors
This command is used to let your OS know that from now on you are using a meta device as your root partition (slice). 09. Check updated /etc/vfstab.
cat /etc/vfstab #device device mount #to mount to fsck point # fd /dev/fd fd no /proc /proc proc no /dev/dsk/c0t0d0s1 swap /dev/md/dsk/d0 /dev/md/rdsk/d0 / swap /tmp tmpfs yes root@rel3 # lockfs fa FS type fsck mount mount pass at boot options
ufs -
no 1
no
While rebooting, the server will be down, but the system would not face any downtime, since two other relay servers are fully operational. 11. After the server reboots, check the status of the root mirror.
root@rel3 # metastat d0: Mirror Submirror 0: d1 State: Okay Pass: 1 Read option: roundrobin (default) Write option: parallel (default) Size: 125510784 blocks (59 GB) d1: Submirror of d0 State: Okay Size: 125510784 blocks (59 GB) Stripe 0: Device Start Block Dbase State Reloc Hot Spare c0t0d0s0 0 No Okay Yes d2: Concat/Stripe Size: 125510784 blocks (59 GB) Stripe 0: Device Start Block Dbase Reloc c0t1d0s0 0 No Yes Device Relocation Information: Device Reloc Device ID c0t1d0 Yes id1,sd@w4849544143484920485553313033303733464c33383030205633575959413941 c0t0d0 Yes id1,sd@w4849544143484920485553313033303733464c33383030205633575958385941
State: Okay Submirror 1: d2 State: Resyncing Resync in progress: 1 % done Pass: 1 Read option: roundrobin (default) Write option: parallel (default) Size: 125510784 blocks (59 GB) d1: Submirror of d0 State: Okay Size: 125510784 blocks (59 GB) Stripe 0: Device Start Block Dbase State Reloc Hot Spare c0t0d0s0 0 No Okay Yes d2: Submirror of d0 State: Resyncing Size: 125510784 blocks (59 GB) Stripe 0: Device Start Block Dbase State Reloc Hot Spare c0t1d0s0 0 No Okay Yes Device Relocation Information: Device Reloc Device ID c0t1d0 Yes id1,sd@w4849544143484920485553313033303733464c33383030205633575959413941 c0t0d0 Yes id1,sd@w4849544143484920485553313033303733464c33383030205633575958385941
Once the syncing is done, d2 submirrors state changes from Resyncing to Okay.
root@rel3 # metastat d0: Mirror Submirror 0: d1 State: Okay Submirror 1: d2 State: Okay Pass: 1 Read option: roundrobin (default) Write option: parallel (default) Size: 125510784 blocks (59 GB) d1: Submirror of d0 State: Okay Size: 125510784 blocks (59 GB) Stripe 0: Device Start Block Dbase State Reloc Hot Spare c0t0d0s0 0 No Okay Yes d2: Submirror of d0 State: Okay Size: 125510784 blocks (59 GB) Stripe 0: Device Start Block Dbase State Reloc Hot Spare c0t1d0s0 0 No Okay Yes Device Relocation Information: Device Reloc Device ID c0t1d0 Yes id1,sd@w4849544143484920485553313033303733464c33383030205633575959413941 c0t0d0 Yes id1,sd@w4849544143484920485553313033303733464c33383030205633575958385941
1.7
Once the syncing is done, d12 submirrors state changes from Resyncing to Okay.
root@rel3 # metastat d10: Mirror Submirror 0: d11 State: Okay Submirror 1: d12 State: Okay Pass: 1 Read option: roundrobin (default) Write option: parallel (default) Size: 16780224 blocks (8.0 GB) d11: Submirror of d10 State: Okay Size: 16780224 blocks (8.0 GB) Stripe 0: Device Start Block Dbase State Reloc Hot Spare c0t0d0s1 0 No Okay Yes d12: Submirror of d10 State: Okay Size: 16780224 blocks (8.0 GB) Stripe 0: Device Start Block Dbase State Reloc Hot Spare c0t1d0s1 0 No Okay Yes
05. Edit /etc/vfstab and update swap related line as to include path to d10 meta device, instead of previously used c0t0d0s1 swap slice.
root@rel3 # cat /etc/vfstab #device device mount FS fsck mount mount
#to mount to fsck point type pass at boot options # fd /dev/fd fd no /proc /proc proc no /dev/dsk/c0t0d0s1 swap no /dev/md/dsk/d0 /dev/md/rdsk/d0 / ufs 1 no swap /tmp tmpfs yes root@rel3 # vi /etc/vfstab (vi is used for file content editing) root@rel3 # cat /etc/vfstab #device device mount FS fsck mount mount #to mount to fsck point type pass at boot options # fd /dev/fd fd no /proc /proc proc no /dev/md/dsk/d10 swap no /dev/md/dsk/d0 /dev/md/rdsk/d0 / ufs 1 no swap /tmp tmpfs yes -
While rebooting, the server will be down, but the system would not face any downtime, since two other relay servers are fully operational. Once the server reboots, the mirroring is done. Procedure used for swap partition mirroring can be used to perform mirroring of other partitions that might be in use. This excludes root partition (procedure previously described) and partitions used for metastat data base. In addition, appropriate eeprom parameter values can be changed (updated), as to enable Solaris to boot even if one of mirrored disks is not available. For more information on this subject, please consult mirroring references, listed in chapter 5.2.
5
5.1
References
Tape Backup
http://docs.sun.com/app/docs/doc/817-1658/6mhcgsubh?a=view#bkuptasks2-26447
5.2
Mirroring
http://www.sun.com/bigadmin/content/submitted/svm_mirroring.jsp http://wikis.sun.com/display/BigAdmin/Mirroring+root+With+Solaris+Volume+Manager +in+the+Solaris+9+and+10+OS http://www.math.uwaterloo.ca/mfcf/internal/procedures/OS/Solaris/mirror.shtml