Networker Cheatsheet | Cyberfella LtdCyberfell...

Search for:

http://www.cyberfella.co.uk/2012/08/28/emc-ne...

Search

Cyberfella Ltd
An I.T. Contractor who writes the useful stu
down.
August 28

Networker Cheatsheet

Here is a handy cheatsheet in troubleshooting failing backups and recoveries

using emcs Networker all taken from real-world experience (and regularly
updated).

1 of 23

08/18/2015 04:21 PM

Networker Cheatsheet | Cyberfella LtdCyberfell...

http://www.cyberfella.co.uk/2012/08/28/emc-ne...

Client cong.
Can you ping the client / resolve the hostname or telnet to 7937?
Are the static routes congured (if necessary).
Can the client resolve the hostnames for the backup interfaces? have
connectivity to them?
Does the backup server appear in the nsr/res/servers le?
Can you run a save -d3 -s /etc on the client?
From the backup server (CLI)
nsradmin -p 390113 -s client
Note: If the name eld is incorrect according to nsradmin (happens when
machines are re-commissioned without being rebuilt) then you need to stop
nsrexecd, rename /nsr/nsrladb folder to /nsr/nsrladb.old, restart nsrexecd,
and most importantly, delete and recreate the client on the networker backup
server, before retrying a savegrp -vc client_name group_name
Also check that all interface names are in the servers le for all interfaces on
all backup servers and storage nodes likely to back the client up.
Can you probe the client?

2 of 23

08/18/2015 04:21 PM

Networker Cheatsheet | Cyberfella LtdCyberfell...

http://www.cyberfella.co.uk/2012/08/28/emc-ne...

savegrp -pvc client groupname

savegrp -D2 -pc client groupname (more verbose)
Bulk import of clients
Instead of adding clients manually one at a time in the NMC, you can
perform an initial bulk import.
nsradmin -i bulk-import-le
where the bulk-import-le contains many lines like this
create type: NSR Client;name:w2k8r2;comment:SOME
COMMENT;aliases:w2k8r2,w2k8r2-b,w2k8r2.cyberfella.co.uk;browse
policy:Six Weeks;retention policy:Six Weeks;group:zzmb-Realign-1;server
network interface:backupsvrb1;storage nodes:storagenode1b1;
Use excel to form a large csv, then use Notepad++ to remove commas. Be
aware there is a comma in the aliases eld, so use an alternative character in
excel to represent this then replace it with a comma once all commas have
been removed from the csv.
Add user to admin list on bu server
nsraddadmin -u user=username, host=*
where username is the username minus the domain name prex (not
necessary).
Starting a Backup / Group from the command line
On the backup server itself: savegrp -D5 -G <group_name>
Ignore the index save sets if you are just testing a group by adding -I
Just backing up the :index savesets in a group: savegrp -O -G <group_name>
On a client: save -s <backup_server_backupnic_name> <path>
Reporting with mminfo
List names of all clients backed up over the last 2 weeks (list all clients)
mminfo -q savetime>2 weeks ago -r client | sort | uniq
mminfo -q client=client-name, level=full -r
client,savetime,ssid,name,totalsize
in a script with a variable, use double quotes so that the variable gets
evaluated, and to sort on american date column
3 of 23

08/18/2015 04:21 PM

Networker Cheatsheet | Cyberfella LtdCyberfell...

http://www.cyberfella.co.uk/2012/08/28/emc-ne...

mminfo -q client=${clientname},level=full -r
client,savetime,ssid,level,volume | sort -k 2.7,2.10n -k 2.1,2.5n -k
2.4,2.5n

mminfo -ot -c client -q savetime>2 weeks ago

mminfo -r ssid,name,totalsize,savetime(16),volume -q
client=client_name,savetime >10/01/2012,savetime <10/16/2012
List the last full backup ssids for subsequent use with recover command
(unix clients)
mminfo -q client=server1,level=full -r client,savetime,ssid
Is the client congured properly in the NMC? (see diagram above for hints
on what to check in what tabs)
How many les were backed up in each saveset (useful for counting les on a
NetApp which is slow using the nd command at host level)
sudo mminfo -ot -q client=mynetappler,level=full,savetime<7 days ago -r
name,nles
name
/my_big_volume

nles
894084

You should probably make use of the ssags option in the mminfo report too,
which adds an extra column regarding the status of the saveset displaying
one or more of the following characters CvrENiRPKIFk with the common
elds shown in bold below along with their meanings.
C Continued, v valid, r purged, E eligible for recycling, N NDMP generated,
i incomplete, R raw, P snapshot, K cover, I in progress, F nished, k
checkpoint restart enabled.
Check Client Index
nsrck -L7 clientname
Backing up Virtual Machines using Networker,VCentre and VADP

4 of 23

08/18/2015 04:21 PM

Networker Cheatsheet | Cyberfella LtdCyberfell...

http://www.cyberfella.co.uk/2012/08/28/emc-ne...

To back up virtual machine disk les on vmfs volumes at the vmware level (as
opposed to the individual le level backups of the individual vms), networker
can interface with the vcenter servers to discover what vms reside on the
esxi clusters managed by them, and their locations on the vmfs shared lun.
For this to work, the shared luns also need to be presented/visible to the
VADP Proxy (Windows server with Networker client and/or Server running as
a storage node) in the fc switch fabric zone cong.

The communication occurs as shown in blue. i.e.

The backup server starts backup group containing vadp clients.
The vadp proxy asks vcentre what physical esxi host has the vm, and where
the les reside on the shared storage luns.
The vadp proxy / networker storage node then tells the esxi host to maintain
a snapshot of the vm while the vmdk les are locked for backup.
the vmdk les are written to the storage device (in my example, a data
domain dedup device)
when the backup is complete, the client index is updated on the backup
server, and the changes logged by the snapshot are applied to the now
unlocked vmdk and then the snapshot is deleted on the esxi host.
Conguring Networker for VADP Backups via a VADP Proxy Storage
Node
The VADP Proxy is just a storage node with bre connectivity to the SAN and
access to the ESXi DataStore LUNs.
In Networker, right click Virtualisation, Enable Auto Discovery

5 of 23

08/18/2015 04:21 PM

Networker Cheatsheet | Cyberfella LtdCyberfell...

http://www.cyberfella.co.uk/2012/08/28/emc-ne...

Complete the elds, but notice there is an Advanced tab. This is to be

completed as follows not necessarily like youd expect

Note that the Command Host is the name of the VADP Proxy, NOT the name
of the Virtual Center Server.
Finally, Run Auto Discovery. A map of the infrastructure should build in the
Networker GUI

Ensure vc, proxy and networker servers all have network comms and can
resolve each others names.
You should now be ready to congure a VADP client.
Conguring a VADP client (Checklist)
GENERAL TAB

6 of 23

08/18/2015 04:21 PM

Networker Cheatsheet | Cyberfella LtdCyberfell...

http://www.cyberfella.co.uk/2012/08/28/emc-ne...

IDENTITY
COMMENT
application_name VADP
VIRTUALIZATION
VIRTUAL CLIENT
(TICK)
PHYSICAL HOST
client_name
BACKUP
DIRECTIVE
VCB DIRECTIVE
SAVE SET
*FULL*
SCHEDULE
Daily Full
APPS AND MODULES TAB

7 of 23

08/18/2015 04:21 PM

Networker Cheatsheet | Cyberfella LtdCyberfell...

http://www.cyberfella.co.uk/2012/08/28/emc-ne...

BACKUP
BACKUP COMMAND
nsrvadp_save -D9
APPLICATION INFORMATION
VADP_HYPERVISOR=fqdn_of_vcenter (hostname in caps)
VADP_VM_NAME=hostname_of_vm (in caps)
VADP_TRANSPORT_MODE=san
DEDUPLICATION
Data Domain Backup
PROXY BACKUP
VMWare
hostname_of_vadp_proxy:hostname_of_vcenter.fqdn(VADP)
GLOBALS 1 OF 2 TAB
ALIASES
hostname
hostname.fqdn
hostname_backup
hostname_backup.fqdn
ip_front
ip_back
GLOBALS 2 OF 2 TAB
REMOTE ACCESS
user=svc_vvadpb,host=hostname_vadp_proxy
user=SYSTEM,host=hostname_vadp_proxy
*@*
OWNER NOTIFICATION
/bin/mail -s client completion : hostname_client nwmonmail
Recovery using recover on the backup client
sudo recover -s backup_server_backup_interface_name
Once in recover, you can cd into any directory irrespective of permissions on
the le system.
Redirected Client Recovery using the command line of the backup
server.
Initiate the recover program on the backup server
sudo recover -s busvr_interface -c client_name -iR -R client_name
or use -iN (No Overwrite / Discard)
-iY (Overwrite)
-iR (Rename ~ )

8 of 23

08/18/2015 04:21 PM

Networker Cheatsheet | Cyberfella LtdCyberfell...

http://www.cyberfella.co.uk/2012/08/28/emc-ne...

Using recover> console

Navigate around the index of recoverable les just like a UNIX lesystem
Recover>

ls

pwd cd\

Change Browsetime
Recover> changetime yesterday
1 Nov 2012 11:30:00 PM GMT
Show versions of a folder or lename backed up
Recover>
versions
(defaults to current folder)
Recover> versions myle
Add a le to be recovered to the list of les to be recovered
Recover> add
Recover>
add myle
List the marked les in the list to be recovered
Recover> list
Show the names of the volumes where the data resides
Recover> volumes
Relocate recovered data to another folder
Recover> relocate /nsr/tmp/myrecoveredles
Recover> relocate E:\\Recovered_Files
Recovery from Linux Svr)

(for Redirected Windows Client

View the folder where the recovered les will be recovered to

Recover> destination
Start Recovery
Recover> recover
VADP Recovery using command line
Prereqs to a successful VADP restore are that the virtual machine be
removed from the Inventory in VCenter (right click vm, remove from
Inventory), and the folder containing the virtual machines les in the vmware
datastore be renamed or removed. If the vm still exists in vmware or in the
datastore, VADP will not recover it.
Log onto the backup server over ssh and obtain the save set ID for your
VADP FULLVM backup.
mminfo avot q name=FULLVM,level=full
Make a note of the SSID for the vm/backup client (or copy it to the cut/paste

9 of 23

08/18/2015 04:21 PM

Networker Cheatsheet | Cyberfella LtdCyberfell...

http://www.cyberfella.co.uk/2012/08/28/emc-ne...

buer)
e.g. 1021210946
Log onto the VADP Proxy (which has SAN connectivity over bre necessary
to recover the les back to the datastore using the san VADP recover mode)
recover.exe S 1021210946 o VADP:host=VC_Svr;VADP:transmode=san
Note that if you want to recover a VM back to a dierent
vCenter,Datastore,ESX host and/or dierent resource pool, you can do that
from the recover command too, rather than waiting to do it using the vsphere
client. this can be used if your vm still exists in vmware and you dont want
to overwrite it. You can additionally specify VADP:host= VADP:datacenter=
VADP:resourcepool= VADP:hostsystem= and VADP:datastore= elds in the
recover command, separated by semicolons and no spaces.
Ive found that whilst the minimal command above may work on some
environments, others demand a far more detailed recover.exe command with
all VADP parameters set before itll communicate with the VC. A working
example is shown below (with each VADP parameter separated on a newline
for readability youll need to put it into a single line, and remove any spaces
between each .
recover.exe -S 131958294 -o
VADP:host=vc.fqdn;
VADP:transmode=san;
VADP:datacenter=vmware-datacenter-name;
VADP:hostsystem=esxihost.fqdn;
VADP:displayname=VM_DISPLAYNAME;
VADP:datastore=cong=VM_DataStore#Hard disk
2=VM_DataStore_LUN_Name#Hard disk 1=VM_DataStore_LUN_Name;
VADP:user=mydomain\vadp_user;
VADP:password=vadp_password
Creating new DataDomain Devices in Networker
In Networker Administrator App from NMC Console, Click Devices button at
the top.
Right click Devices in the Left hand pane, New Device Wizard (shown)

10 of 23

08/18/2015 04:21 PM

Networker Cheatsheet | Cyberfella LtdCyberfell...

http://www.cyberfella.co.uk/2012/08/28/emc-ne...

Select Data Domain, Next, Next

Use an existing data domain system

Choose a data domain system in the same physical location to your backup
server!
Enter the Data Domain OST username and password

Browse and Select

Create a New Folder in sequence, e.g. D25, tick it.

11 of 23

08/18/2015 04:21 PM

Networker Cheatsheet | Cyberfella LtdCyberfell...

http://www.cyberfella.co.uk/2012/08/28/emc-ne...

Highlight the automatically generated Device Name, Copy to clipboard

(CTRL-C), Next

Untick Congure Media Pools (label device afterwards using Paste from
previous step), Next

12 of 23

08/18/2015 04:21 PM

Networker Cheatsheet | Cyberfella LtdCyberfell...

http://www.cyberfella.co.uk/2012/08/28/emc-ne...

Select Storage Node to correspond with device locality from Use an existing
storage node, Next
Agree to the default SNMP info (unless reconguration for custom
monitoring environment is required), Next
Congure, Finish
Select new device (unlabelled, Volume name blank), right click, Label

Paste Device Name in clipboard buer (CTRL-V)

Select Pool to add the Device into, OK.

Slow backups of large amounts of data to DataDomain deduplication

device
If you have ridiculously slow backups of large amounts of data, check in
Networker NMC to see the name of the storage node (Globals2 tab of the
client conguration), then connect to the DataDomain and look under the
Data Management, DD Boost screen for Clients of which your storage node
will be one. Check how many CPUs and Memory it has. e.g. Guess which

13 of 23

08/18/2015 04:21 PM

Networker Cheatsheet | Cyberfella LtdCyberfell...

http://www.cyberfella.co.uk/2012/08/28/emc-ne...

one is the slow one (below)

Then SSH to the storage node and check what processes are consuming the
most CPU and Memory (below)

In this example (above), despite dedicating a storage node backup a single

large applications data, the fact that it only has 4 cpus and is scanning every

14 of 23

08/18/2015 04:21 PM

Networker Cheatsheet | Cyberfella LtdCyberfell...

http://www.cyberfella.co.uk/2012/08/28/emc-ne...

le that ddboost is attempting to deduplicate means that a huge bottleneck is

introduced. This is a typical situation whereby decommissioned equipment
has been re-purposed.
Networker Server
ssh to the networker server and issue the nsrwatch command. Its a
command line equivalent to connecting to the Enterprise app in the NMC
and looking at the monitoring screen. Useful if you cant connect to the
NMC.

Blank / Empty Monitoring Console

If youre NMC is displaying a blank monitoring console, try this before
restarting the NMC
Tick or Un-tick and Re-tick Archive Requests.

15 of 23

08/18/2015 04:21 PM

Networker Cheatsheet | Cyberfella LtdCyberfell...

http://www.cyberfella.co.uk/2012/08/28/emc-ne...

Tape Jukebox Operations

ps -ef | grep nsrjb
-Maybe necessary to kill o any pending nsrjb processes
before new ones will work.
nsrjb -C | grep <volume>
nsrjb -w -S <slot>

-Identify the slot that contains the tape (volume)

-Withdraw the tape in slot <slot>

nsrjb -d
-Deposit all tapes in the cap/load port into empty slots in the
jukebox/library.
Note: If you are removing and replacing tapes you should take note what
pools the removed tapes belong it and allocate new blank tapes deposited
16 of 23

08/18/2015 04:21 PM

Networker Cheatsheet | Cyberfella LtdCyberfell...

http://www.cyberfella.co.uk/2012/08/28/emc-ne...

into the library to the same pools to eliminate impact on backups running out
of tapes.
Exchange Backups
The application options of the backup client (exchange server in DAG1 would
be as follows
NSR_SNAP_TYPE=vss
NSR_ALT_PATH=C:\temp
NSR_CHECK_JET_ERRORS=none
NSR_EXCH2010_BACKUP=passive
NSR_EXCH_CHECK=no
NSR_EXCH2010_DAG=GB-DAG1
NSR_EXCH_RETAIN_SNAPSHOTS=no
NSR_DEVICE_INTERFACE=DATA_DOMAIN
NSR_DIRECT_ACCESS=no
Adding a NAS lesystem to backup (using NDMP)
Some pre-reqs on the VNX need to be satised before NDMP backups will
work. This is explained here
http://www.cyberfella.co.uk/2013/12/03/congure-vnx-for-ndmp/

General tab

17 of 23

08/18/2015 04:21 PM

Networker Cheatsheet | Cyberfella LtdCyberfell...

http://www.cyberfella.co.uk/2012/08/28/emc-ne...

The exported fs name can be determined by logging onto the VNX

as nasadmin and issuing the following command
server_mountpoint server_2 -list
Apps and Modules tab

18 of 23

08/18/2015 04:21 PM

Networker Cheatsheet | Cyberfella LtdCyberfell...

http://www.cyberfella.co.uk/2012/08/28/emc-ne...

Application Options that have worked in testing NDMP Backups.

Leave datadomain unticked in Networker 8.x and ensure youve selected a
device pool other than default, or Networker may just sit waiting for a tape
while youre wondering why NDMP backups arent starting!
HIST=y
UPDATE=y
DIRECT=y
DSA=y
SNAPSURE=y
#OPTIONS=NT
#NSR_DIRECT_ACCESS=NO
#NSR_DEVICE_INTERFACE=DATA_DOMAIN
Backup Command: nsrndmp_save -s backup_svr -c nas_name -M -T vbb -P

19 of 23

08/18/2015 04:21 PM

Networker Cheatsheet | Cyberfella LtdCyberfell...

http://www.cyberfella.co.uk/2012/08/28/emc-ne...

storage_node_bu_interface or dont use -P if Backup Server acts as SN.

To back up an NDMP client to a non-NDMP device, use the -M option.
The value for the NDMP backup type depends on the type of NDMP host.
For example, NetApp, EMC, and Procom all support dump, so the value for
the Backup Command attribute is:
nsrndmp_save -T dump
Globals 1 tab

Globals2 tab

20 of 23

08/18/2015 04:21 PM

Networker Cheatsheet | Cyberfella LtdCyberfell...

http://www.cyberfella.co.uk/2012/08/28/emc-ne...

Important: If the lesystem being backd up contains more than 5 million

les, set the timeout attribute to zero in the backup groups properties.
Command line equivalent to the NMCs Monitoring screen
nsrwatch
Command line equivalent to the NMCs Alerts pane
printf show pending\nprint type:nsr\n | /usr/sbin/nsradmin -iResetting Data Domain Devices
Running this in one go if youve not done it before is not advised. Break it up
into individual commands (separated here by pipes) and ensure the output is
what youd expect, then re-join commands accordingly so youre certain
youre getting the result you want. This worked in practice though. It will
only reset Read Only (.RO) devices so it wont kill backups, but will
potentially kill recoveries or clones if they are in progress.
nsr_render_log -lacedhmpty -S 1 hour ago /nsr/logs/daemon.raw | grep -i
critical | grep RO | awk {print $10} | while read eachline; do nsrmm | grep
$eachline | cut -d, -f1 | awk {print $7}; done | while read eachdevice; do
nsrmm -HH -v -y -f ${eachdevice}; done

Posted August 28, 2012 by Matthew Bradley in category Backup / Disaster

Recovery, VMWare / Cloud Computing
Post navigation
Create lesystem on emc VNX Tuning an SSD powered Linux PC

21 of 23

08/18/2015 04:21 PM

Networker Cheatsheet | Cyberfella LtdCyberfell...

http://www.cyberfella.co.uk/2012/08/28/emc-ne...

Availability & CV
Company & Contact Details
Cyberfella Ltd. is born

Areas of Expertise
Areas of Expertise Select Category

Tip Jar
If I've helped you out of a pinch and saved your project time and money and
you wish to thank me, feel free to donate some coins to the tip jar via PayPal.

Dell BIOS updates w/o Windows

ISL conguration on Cisco MDS
Zone conguration on Cisco MDS
Share all subfolders as individual hidden shares
Installing Dropbox on ElementaryOS (Freya)
Finding SIDs that havent changed after SetACL
Using Linux commands on WIndows
Comparing text les using Notepad++
Re-ACL of inter-domain migrated data
View Windows Permissions / ACLs

From days gone by

22 of 23

08/18/2015 04:21 PM

Networker Cheatsheet | Cyberfella LtdCyberfell...

http://www.cyberfella.co.uk/2012/08/28/emc-ne...

From days gone by Select Month

82,689 spam

blocked by Akismet
Cyberfella Ltd. Established 14th February 2012. Company Number. 7950008,
VAT Registered.
Theme "Anarcho Notepad" designed and engineered by Arthur (Berserkr)
Gareginyan.

23 of 23

08/18/2015 04:21 PM