Вы находитесь на странице: 1из 33

Sun ZFS Storage Appliance Recovery - Practice 1

Lab Scenario
BARNABAS paycheck services was running the weekly payroll report when all of sudden the
7410 ZFS appliance that stores all of their payroll data lost one of its clustered NAS heads.
Since the 7410 was in a clustered configuration it will continue to run with the single NAS head
handling all traffic, but at a performance cost. BARNABAS must deliver the payroll report on
time to their customer or they will lose their contract.

You have been dispatched to restore the 7410 NAS head and return it to a online clustered
state, BARNABAS paycheck services is depending on your technical knowledge and skill.
Good Luck!

Make sure you have the FISHsitck that matches the system type.
Never do this procedure without an L2 TSC action plan. Remember this is a last resort
operation.

Procedure Summary
1. UNCONFIG the Cluster

2. FISHSTICK the 7410 NAS head

3. Restart Machine from ILOM

4. Go Into BIOS and Change Boot Order

5. Booting from FISHSTICK

6. Go Into BIOS and Revert Boot Order

7. Set Up Basic Network Configuration to Use BUI.

8. Updating the System with the latest version of the Software

9. Updating the Software from the BUI

  Copyright © 2012, Oracle and/or its affiliates. All rights reserved.   
Sun ZFS Storage Appliance Recovery Practice
Effective Page 1 of 33 Rev
Lab Environment
iLOM / Service Login and
System Name BUI IP
Processor IP password
to8.amber.com (Top 7410) https://10.7.6.81:215 https://10.7.6.80
to9.amber.com (Bottom 7410) https://10.7.6.91:215 https://10.7.6.90 Root
fu6.amber.com (Top 7210) https://10.7.6.61:215 https://10.7.6.60 changeme
fu7.amber.com (Bottom 7210) https://10.7.6.71:215 https://10.7.6.70

  Copyright © 2012, Oracle and/or its affiliates. All rights reserved.   
Sun ZFS Storage Appliance Recovery Practice
Effective Page 2 of 33 Rev
Un-configure the Cluster
1. Log on to the active 7410 NAS head using a pc with a compatible web browser. Use the IP
address of the NAS head and long on:
 Example: https://10.7.6.81:215
 User = root
 Password = changeme

a. Go to Configuration then click on Cluster, at the Cluster screen click the UNCONFIG
button.

  Copyright © 2012, Oracle and/or its affiliates. All rights reserved.   
Sun ZFS Storage Appliance Recovery Practice
Effective Page 3 of 33 Rev
b. The following splash screen appears, read the message, then click OK. By performing
the UNCONFIG, this action will keep the NAS from trying to form a cluster once the
defective NAS head has been restored. The two NAS heads are re-clustered later in the
lab. Ask your instructor if you have any question about the UNCONFIG.

c. After the UNCONFIG, notice the state of the pools and the SETUP button is active.

  Copyright © 2012, Oracle and/or its affiliates. All rights reserved.   
Sun ZFS Storage Appliance Recovery Practice
Effective Page 4 of 33 Rev
FISHSTICK the 7410 NAS head
What is FISHSTICK? FISHSTICKs are bootable USBs for reinstalling SS7000 series
software. Going through this process WILL DESTROY customer configurations unless you have
back up or the node you are trying to recover is one of the nodes in cluster.

2. Ask the instructor the FISHSTICK USB stick. (For this lab the FISHSTICK USB image has
been created ahead of time) Make sure you have the FISHSTICK that matches the system
type. Never do this procedure without an L2 TSC action plan. Place the USB stick in one of
USB ports on the front of the defective 7410 NAS head

Note: Remember this is a last resort operation.

  Copyright © 2012, Oracle and/or its affiliates. All rights reserved.   
Sun ZFS Storage Appliance Recovery Practice
Effective Page 5 of 33 Rev
Restart Machine from ILOM
3. Connect to ILOM via the network using putty.exe. Type the IP address of the ILOM
10.7.6.90 then click Open.

a. At the PuTTY window, type the following.


 Login = root
 Password = changme
 Then press Enter.

  Copyright © 2012, Oracle and/or its affiliates. All rights reserved.   
Sun ZFS Storage Appliance Recovery Practice
Effective Page 6 of 33 Rev
b. Type the following command to stop and start the SP console.
Note: SP console may be running (not usually), by using the stop /SP/console command,
insures SP console is not running.
 stop /SP/console
 start /SP/console
 Then answer "y", SP console will start.

c. Press the Power button on the front of the NAS head, monitor the PuTTY session.

  Copyright © 2012, Oracle and/or its affiliates. All rights reserved.   
Sun ZFS Storage Appliance Recovery Practice
Effective Page 7 of 33 Rev
Go Into BIOS and Change Boot Order
4. Right after the boot, you will see start up screen such as above. After you see "01 USB
Massive Device Detected", press CTRL and E key at the same time. (CTRL-E) After a while
you should see the screen as below. It is the top menu of the BIOS. This will take a few
seconds.

a. Use right cursor key to move to [Boot] menu. You should see the screen such as below.

  Copyright © 2012, Oracle and/or its affiliates. All rights reserved.   
Sun ZFS Storage Appliance Recovery Practice
Effective Page 8 of 33 Rev
b. Press down cursor key to choose [Hard Disk Drive] menu, then press Enter.

c. Use the arrow keys and select 1st Drive, and then press Enter. At the options menu,
use the arrow keys and select the USB device, then press Enter.

  Copyright © 2012, Oracle and/or its affiliates. All rights reserved.   
Sun ZFS Storage Appliance Recovery Practice
Effective Page 9 of 33 Rev
Note: Notice the USB device is the 1st Drive.

d. Now, press [ESC] to leave this menu. Keep pressing a right arrow key until the [EXIT]
menu is selected. Confirm the highlight is on "Save Changes and Exit." By pressing
[enter] it will ask you to confirm saving the changes press Enter again to reboot the box.
It may take 30 to 60 seconds.

  Copyright © 2012, Oracle and/or its affiliates. All rights reserved.   
Sun ZFS Storage Appliance Recovery Practice
Effective Page 10 of 33 Rev
Booting from FISHSTICK
5. Booting from FISHSTICK is no different from any operating system boot.

Warning: DO NOT touch anything unless the last prompt comes up.

Look at the following screen shots you should see something similar, if you do not ask your
instructor for assistance.

  Copyright © 2012, Oracle and/or its affiliates. All rights reserved.   
Sun ZFS Storage Appliance Recovery Practice
Effective Page 11 of 33 Rev
  Copyright © 2012, Oracle and/or its affiliates. All rights reserved.   
Sun ZFS Storage Appliance Recovery Practice
Effective Page 12 of 33 Rev
a. Automatic Recovery Runs: As seem before, the installation runs automatically. At the
end of this installation, you are prompted to remove the USB stick and press any key to
reboot.

b. Remove the USB from SS7000 series and press any key to reboot the box.

  Copyright © 2012, Oracle and/or its affiliates. All rights reserved.   
Sun ZFS Storage Appliance Recovery Practice
Effective Page 13 of 33 Rev
Go Into BIOS and Revert Boot Order
6. After the installation, you will need to go back to the BIOS by following Step-2 above.
Ensure that the boot order has reverted to the same order in Step-2.
a. If you did not make note of the boot order at Step -2. Most of the time, the boot order
should be
 1st Drive [SATA:11M-XXXXXX]
 2nd Drive [SATA:12M-XXXXXX]

b. Verify boot order then press use ESC to exit the menu then press ESC again, at the
Discard changes and exit screen select OK and press Enter, the system starts to boot.

  Copyright © 2012, Oracle and/or its affiliates. All rights reserved.   
Sun ZFS Storage Appliance Recovery Practice
Effective Page 14 of 33 Rev
Set Up Basic Network Configuration to Use BUI
7. You will be prompted with the same message you get at initial installation. Follow the
installation guide to set up IP address for configuring from the browser. Press any key to
begin configuration.

a. Use the configuration data recorded earlier in lab to configure the appliance.
 Use the arrow keys to change fields.
 Use the Delete key to remove unwanted data.

Note: Sometimes, data may exist in the various fields and you can delete it if necessary.

 Press enter to input information. Notice the cursor advance to the next line. Make
sure you keep the password set to changeme. Do not use any other password.
 Host Name: to9
 DNS Domain: amber.com
 IP Address: 10.7.6.91
 IP Netmask: 255.255.0.0
 Default Router: 10.7.0.1
 DNS Server: 10.7.6.50
 Password: changeme
 Re-enter Password: changme
 

  Copyright © 2012, Oracle and/or its affiliates. All rights reserved.   
Sun ZFS Storage Appliance Recovery Practice
Effective Page 15 of 33 Rev
b. After you have entered all of the information, verify it, and then press ESC – 1.
 The puTTY screen should display the following after 15 to 30 seconds:

  Copyright © 2012, Oracle and/or its affiliates. All rights reserved.   
Sun ZFS Storage Appliance Recovery Practice
Effective Page 16 of 33 Rev
Updating the System with the latest version of the Software
8. After the system has been restored, you need to bring the system back to the latest version
of the appliance kit. In some cases, you will be instructed to bring the system up to the last
version that the customer was running. In other cases, you will be instructed to bring the
system up to the lasted version.

a. Open a browser and type in the IP address of the FISKstick'd NAS head.

 Example: https://10.7.6.91:215
 User = root
 Password = changeme

b. The Welcome screen appears, click the Start button, "Do not configure anything on the
FISKstick'd NAS head", the NAS head will be configured when the two NAS head are
re-clustered. Click the SKIP button on the Configure Clustering.

  Copyright © 2012, Oracle and/or its affiliates. All rights reserved.   
Sun ZFS Storage Appliance Recovery Practice
Effective Page 17 of 33 Rev
c. On the Configure Networking and Configure DNS screens, click the COMMIT button.

d. Click Commit on the Configure Time and Configure Name Services screens.

e. At the Configure Storage screen COMMIT button, then at the Registration & Support
screen, click the LATER button. When the Register Later screen appears click OK.

  Copyright © 2012, Oracle and/or its affiliates. All rights reserved.   
Sun ZFS Storage Appliance Recovery Practice
Effective Page 18 of 33 Rev
Updating the Software from the BUI
9. At the BUI go to Maintenance then SYSTEM.

a. Click on the + Icon next to Available Updates then the Add Software Update screen
appears, click Browse. Select the code provided to you by your instructor; it should be
on a USB stick.

Example: ak-nas-2011-04-24-3-0-1-1-19-nd.pkg.gz

b. Click the UPLOAD button.

  Copyright © 2012, Oracle and/or its affiliates. All rights reserved.   
Sun ZFS Storage Appliance Recovery Practice
Effective Page 19 of 33 Rev
c. A status bar appears while the software uploads; it may take 5 to 10 minutes.

d. The update file will upload and then unpack automatically.

  Copyright © 2012, Oracle and/or its affiliates. All rights reserved.   
Sun ZFS Storage Appliance Recovery Practice
Effective Page 20 of 33 Rev
e. Once the file is unpacked, you are now ready to start the update. Click the right arrow icon next
to the software update.

f. A warning display stating that a reboot is required. Click on the APPLY button.

g. This will take several minutes. When the updates are finished, the system will reboot

  Copyright © 2012, Oracle and/or its affiliates. All rights reserved.   
Sun ZFS Storage Appliance Recovery Practice
Effective Page 21 of 33 Rev
h. As the system reboots the following screen is displayed.

i. Log on to the FISKstick'd NAS head, then verify the Current System software.

  Copyright © 2012, Oracle and/or its affiliates. All rights reserved.   
Sun ZFS Storage Appliance Recovery Practice
Effective Page 22 of 33 Rev
j. Click on the FACTORY RESET button, and then click OK. A factory reset must be
performed before the two NAS head are re-clustered. Any questions ask your
instructor.

k. The system will perform a factory resent and reboot.

  Copyright © 2012, Oracle and/or its affiliates. All rights reserved.   
Sun ZFS Storage Appliance Recovery Practice
Effective Page 23 of 33 Rev
l. Take a look at the FISHSTICK'd NAS head PuTTY session, notice the message "Press
any key to begin configuring appliance", leave it in the following state, do not attempt to
configure the NAS head.

m. Log on to the other NAS head, go to the Configuration >>> CLUSTER screen, click on
SETUP.

  Copyright © 2012, Oracle and/or its affiliates. All rights reserved.   
Sun ZFS Storage Appliance Recovery Practice
Effective Page 24 of 33 Rev
n. Verify the cluster cabling then at the Cable Cluster screen click the COMMIT button.

o. At the Set Name and Password screen type Appliance Name, Root Password and
Confirm Password then click COMMIT.
 Appliance Name = to9
 Root Password = changeme
 Confirm Password = changeme

  Copyright © 2012, Oracle and/or its affiliates. All rights reserved.   
Sun ZFS Storage Appliance Recovery Practice
Effective Page 25 of 33 Rev
p. Take a look at the upper right hand corner, notice the messages and status bar.

q. Both NAS heads are now Clustered, at the very top of the BUI screen, there is a
message stating, "The cluster peer has rejoined the cluster".

  Copyright © 2012, Oracle and/or its affiliates. All rights reserved.   
Sun ZFS Storage Appliance Recovery Practice
Effective Page 26 of 33 Rev
Optional Labs
Hardware Replacement Labs - Practice 2
Lab Scenario
The student should know how to perform the following repair and replacement tasks on the ZSF 7000 

The student will review this list and perform the as many tasks they feel necessary. These replacement 
procedures are referenced MOS. So the student should feel comfortable in carrying out any of this 
Action Plans on a 7000 appliance 

The student can take both 7410 system apart so as long as they are both restored to the original 
clustered configuration. The student should treat this equipment as if it the customers equipment. 
Follow safety precautions ESD etc. The student should be encouraged to film or take pictures of the 
system as they got through the physical repair tasks. 

1401562.1 How to Replace a System Disk in a Sun ZFS Unified Storage Appliance
7310 or 7410 :ATR:1401562.1:0
https://support.us.oracle.com/oip/faces/secure/km/DocumentDisplay.jsp
x?id=1401562.1
1421284.1 How To Replace a Sun ZFS Unified Storage Appliance 7000 PCI Riser
Board:ATR:1421284.1:
https://support.us.oracle.com/oip/faces/secure/km/DocumentDisplay.jsp
x?id=1421284.1
1383143.1 How to Replace a Memory DIMM in Sun ZFS Unified Storage Appliance
7110, 7310 and 7410 Appliance:ATR:1383143.1:0
https://support.us.oracle.com/oip/faces/secure/km/DocumentDisplay.jsp
x?id=1383143.1
1382344.1 How to replace the fan power-board on an Sun ZFS Unified Storage
Appliance 7110, 7310 or 7410 system:ATR:1382344.1:3
https://support.us.oracle.com/oip/faces/secure/km/DocumentDisplay.jsp
x?id=1382344.1
1381523.1 How To Replace A PSU In The J44x0 And J4500 JBOD Attached To
Sun ZFS Unified Storage Appliance 7x10 And 7x20 Standalone Or
Cluster.:ATR:1381523.1:0
https://support.us.oracle.com/oip/faces/secure/km/DocumentDisplay.jsp
x?id=1381523.1
1380155.1 How To Remove And Replace A SAS Or SAS-2 Cable On The ZFS
Unified Storage Appliance S7x10 And S7x20 NAS With
JBODs:ATR:1380155.1:
https://support.us.oracle.com/oip/faces/secure/km/DocumentDisplay.jsp
x?id=1380155.1
1377473.1 How to replace a failed fan module in Sun ZFS Unified Storage
Appliance 7000-systems:ATR:1377473.1:0
https://support.us.oracle.com/oip/faces/secure/km/DocumentDisplay.jsp
x?id=1377473.1

  Copyright © 2012, Oracle and/or its affiliates. All rights reserved.   
Sun ZFS Storage Appliance Recovery Practice
Effective Page 27 of 33 Rev
1375501.1 How to Replace a data drive in a J44X0-JBOD attached to Sun ZFS
Unified Storage Appliance 7310, 7410, 7120,7320, and
7420:ATR:1375501.1:0
https://support.us.oracle.com/oip/faces/secure/km/DocumentDisplay.jsp
x?id=1375501.1
1357354.1 Sun Storage 7000 Unified Storage System: How to update the BIOS for
a 7x10 NAS appliance
https://support.us.oracle.com/oip/faces/secure/km/DocumentDisplay.jsp
x?id=1357354.1
1333116.1 Sun Storage 7000 Unified Storage System: How to Apply Deferred
Updates
https://support.us.oracle.com/oip/faces/secure/km/DocumentDisplay.jsp
x?id=1333116.1
1329307.1 Sun Storage 7000 Unified Storage System: How to set up NAS
clustering
https://support.us.oracle.com/oip/faces/secure/km/DocumentDisplay.jsp
x?id=1329307.1
1396214.1 How To Remove And Replace A ZFSSA Unified Storage 7410 And 7410
Cluster Power Distribution Board.:ATR:1396214.1:3
https://support.us.oracle.com/oip/faces/secure/km/DocumentDisplay.jsp
x?id=1396214.1
1382257.1 How to Replace Sun ZFS Unified Storage Appliance 7410/7410 CPU
Module & Heat Sink:ATR:1382257.1:3
https://support.us.oracle.com/oip/faces/secure/km/DocumentDisplay.jsp
x?id=1382257.1
1382512.1 How to replace Sun ZFS Unified Storage Appliance CPU Module & Heat
Sink:ATR:1382512.1:3
https://support.us.oracle.com/oip/faces/secure/km/DocumentDisplay.jsp
x?id=1382512.1
1386224.1 How to replace a PCIe-card in a ZFSSA Unified Storage
System:ATR:1386224.1:2
https://support.us.oracle.com/oip/faces/secure/km/DocumentDisplay.jsp
x?id=1386224.1
1391828.1 Sun Storage 7000 Unified Storage System: How to Replace a 7x10
Appliance System Board
https://support.us.oracle.com/oip/faces/secure/km/DocumentDisplay.jsp
x?id=1391828.1
1393597.1 How To Replace a Sun ZFS Unified Storage Appliance 7310/7410 PCI
Riser Board:ATR:1393597.1:0
https://support.us.oracle.com/oip/faces/secure/km/DocumentDisplay.jsp
x?id=1393597.1
1410463.1 How to Replace Sun ZFS Unified Storage Appliance 7000 Hard Disk
Drive:ATR:1410463.1:0
https://support.us.oracle.com/oip/faces/secure/km/DocumentDisplay.jsp
x?id=1410463.1
   
  Copyright © 2012, Oracle and/or its affiliates. All rights reserved.   
Sun ZFS Storage Appliance Recovery Practice
Effective Page 28 of 33 Rev
1408348.1 How to Replace a Sun ZFS Unified Storage Appliance Memory
DIMM:ATR:1408348.1:0
https://support.us.oracle.com/oip/faces/secure/km/DocumentDisplay.jsp
x?id=1408348.1
1407494.1 How To Replace A PSU In The Sun ZFS Unified Storage Appliance
Standalone Or Cluster:ATR:1407494.1:0
https://support.us.oracle.com/oip/faces/secure/km/DocumentDisplay.jsp
x?id=1407494.1
1401562.1 How to Replace a System Disk in a Sun ZFS Unified Storage Appliance
7310 or 7410 :ATR:1401562.1:0
https://support.us.oracle.com/oip/faces/secure/km/DocumentDisplay.jsp
x?id=1401562.1

  Copyright © 2012, Oracle and/or its affiliates. All rights reserved.   
Sun ZFS Storage Appliance Recovery Practice
Effective Page 29 of 33 Rev
Obtaining the Latest Version of the Appliance Kit - Practice 3
The FISHSTICK does not always have the latest version of the Appliance Kit Software or the
version that your customer’s might be using. It is very important that you get customer back to
the current version of the Appliance Kit or to the version they are running.
For this lab, you will assume that the action plan states to bring the customer up to the latest
version of the appliance kit software.

Go to https://wikis.oracle.com/display/FishWorks/Fishworks, and then click on Software


Updates. Note you will need VPN access for this URL. Remember to review the release notes.
Download the latest version to your laptop. You need it later in the lab 1b step eight. Once the
file downloaded you will need to unzip it.
 

  Copyright © 2012, Oracle and/or its affiliates. All rights reserved.   
Sun ZFS Storage Appliance Recovery Practice
Effective Page 30 of 33 Rev
Create a FISHSTICK for a 7000 - Practice 4
The lab will provide the student with the instructions on how to create a FISHSTICK of their
own. However do to time constraints; the FISHSTICK will be made in advance for the
appropriate appliance.The Following are the FISHSTICKS for the lab equipment:

Code Available Number of


Model
Name Versions FISHSTICKS

7210 Fugu 2010.Q3.4.0 2

7410 Toro 2010.Q3.4.0 2

LAB Notes:
This Lab will have in bold letters that states the FISHSTICK procedure is on to be done if and
only if a L2 TSE documents it in an action plan and this procedure is a last resort.

Students will use one of the four premade USB FISHSTICKS to perform the FISHSTICK
operation on the lab equipment.
Below are the procedures to that are documents for creating a FISHSTICK.

FISHSTICK Creation Procedure


To create a FISHSTICK takes about 30 minutes, This is a reading only lab to give them
procedure on how to create a FISHSTICK in the field. Below is the FISHSTICK creation
documentation.

What is FISHSTICK?
FISHSTICKs are bootable USBs for reinstalling SS7000 series software. Going through this
process WILL DESTROY customer configurations unless you have back up or the node you are
trying to recover is one of the nodes in cluster.
 

  Copyright © 2012, Oracle and/or its affiliates. All rights reserved.   
Sun ZFS Storage Appliance Recovery Practice
Effective Page 31 of 33 Rev
Requirements:
Putty or other terminal emulator for laptop
1x 1Gb or more USB memory stick
1x x86 Solaris system running Solaris 10 or later
mkusb tool
Appropriate ISO image:
2010.Q3.4.0-XXX.iso where XXX codename translates to product name:

Product Name Code Name


7110 iwashi
7210 fugu
7310 maguro
7410 toro
7420 otoro
7120 iwashi_plus
7320 maguro_plus

Procedure
1. FISHSTICK images can be found here: 
/net/nfssys.uk/export/install/unbundled/storage/amber-
road/FISHSTICKs
2. Get mkusb(info) tool and appropriate image for your product from the table above.
3. from here on the cores systems - /net/nfssys.uk/export/install/unbundled/storage/amber-
road/FISHSTICKs/mkusb
4. Insert the USB stick into the Solaris box
- if you are running this on Solaris 10, you must do the following:
# svcs -a | grep volfs
online 19:17:54 svc:/system/filesystem/volfs:default
# svcadm disable volfs
# svcs -a | grep volfs
disabled 19:18:23 svc:/system/filesystem/volfs:default
5. On Solaris 10 or opensolaris, you must also make sure the USB drive is unmounted.
On newer versions of Solaris (ie try this and if it fails on Solaris 10 then don't worry), run
"eject -l" as the stick is most likely now mounted and needs to be unmounted, eg:
# eject -l
/dev/dsk/c10t0d0s0 rmdisk1,/media/DataTraveler 2.0
/dev/dsk/c2t0d0s2 cdrom1,cd1,sr1
/dev/dsk/c7t0d0s2 cdrom,cdrom0,cd,cd0,sr,sr0
# eject rmdisk1
rmdisk1 /dev/dsk/c10t0d0s0 unmounted
 

  Copyright © 2012, Oracle and/or its affiliates. All rights reserved.   
Sun ZFS Storage Appliance Recovery Practice
Effective Page 32 of 33 Rev
6. Run 'rmformat -l' to find out the correct device path to the USB stick
# rmformat -l
Looking for devices...
1. Logical Node: /dev/rdsk/c10t0d0p0
Physical Node:
/pci@0,0/pci1022,7460@6/pci1022,7464@0,1/hub@3/storage@2/disk@0,0
Connected Device: Kingston DataTraveler 2.0 1.00
Device Type: Removable
Bus: USB
Size: 3.8 GB
Label: <None>
Access permissions: Medium is not write protected.
...

7. Run 'rmformat -l' to find out the correct device path to the usb stick
8. Run 'mkusb file.iso /dev/rdsk/<usb>p0'
* You might need to run the mkusb with the full path specified.
* Replace file.iso with the iso image file corresponding to the product you are going to install.
It might take about 30 minutes to complete.
9. Re-enable volfs on Solaris 10 if you disabled it:
# svcs -a | grep volfs disabled 19:16:12
svc:/system/filesystem/volfs:default
# svcadm enable volfs
# svcs -a | grep volfs online 19:17:54
svc:/system/filesystem/volfs:default
10. Know you ready to insert the stick into the 7xx0 system.

  Copyright © 2012, Oracle and/or its affiliates. All rights reserved.   
Sun ZFS Storage Appliance Recovery Practice
Effective Page 33 of 33 Rev