Isilon NDMP backup failed after NetWorker upgraded to 19.11

We need to update firmware of existing Isilon to 9.7.1.x to add new Isilon nodes as recommended by support.  Before we do that, we need to confirm about NDMP backup with Isilon.  Because we have NDMP backup of Isilon by NetWorker, NetWorker upgrade to 19.11 from 19.10.0.4 is required.  After NetWorker upgrade in Feb, some of the NDMP backups failed with error message "Hostname resolution failed ".   Multiple retries will work.  However, it is very annoying.  

After working with support, there is new change to NetWorker in 19.11.  See kb NetWorker: server upgraded to 19.11, backup fails reporting "Hostname resolution failed" | Dell US

None of the workaround in the kb above will fix the issue on Isilon.  The forward DNS lookup for Isilon are actually forwarded to Smartconnect SIP from DNS server.  Only option is to add a reverse entries to the DNS server for all the Isilon nodes that handle NDMP backup.  Refer to link SmartConnect and Reverse DNS | Dell PowerScale: Network Design Considerations | Dell Technologies Info Hub.  

So, pointer records for all Isilon nodes handling NDMP backup will be created on the DNS server for the NDMP zone name.  Once that is done, all backup is fine.  Keep in mind there is no change on the forward lookup which is still handled by Smartconnect SIP.  Not sure if there is a fix now.  

NetWorker NDMP backup for Isilon failed after upgrade from 19.4.0.1 to 19.4.0.3

All NDMP backup for Isilon failed after upgraded from 19.4.0.1 to 19.4.0.3. There is no change and firmware update on Isilon side. From NetWorker log, handshake is fine, and backup seems to start. 

42617:nsrndmp_save: NDMP Service Log: 

Filetransfer: Transferred 1271296 bytes in 0.125 seconds throughput of 9969.966 KB/s 

Total 1271296 bytes in this backup stream 

However, the log shows error below. When I monitor from Isilon, I don't see read activity after first few min. 

42923:nsrndmp_save: NDMP Service Error: Failed to write data at offset 220672 

42738:nsrndmp_save: Data server halted: Error during the backup. 

The error seems to point to write issue on DataDomain side. However, all other backup is fine writing to the same DD. The workaround is to disable client direct and use the old trick to use -P option to specify the storage node for NDMP backup command in NetWorker.

==================================================================== 

Forget to mention: we have 2 sites with the same upgrade. The other site works fine. Only difference between them is multi-home. The site with issue is multihome. The site that works fine after upgrade setup with dedicated backup network.

To check from Isilon side, look for NDMP logs in /ifs/.ifsvar/modules/ndmp/trade/node<ID>-ndmpd.trace

Backup Unix raw device with NetWorker

To backup Unix Raw Device in NetWorker, pls follow esg59701 or video in EMC community


Here is the kb article from EMC website.

How can NetWorker be configured to backup a rawdevice?

How to backup a RAW device?

 

Fact

NetWorker for UNIX/Linux 7.x.x

 

Resolution

Create a directive using rawasm. Rawasm used to backup /dev entries (block andcharacter-special files) and their associated raw disk partitiondata. On some systems, /dev entries are actually symboliclinks to device specific names. Unlike other ASMs, this ASM followssymlinks, allowing the shorter /dev name to be configured.

1. Open NMC.
2. Go to the Configuration > Directives.
3. Right Click > new.
4. In the Name field enter Raw.
5. In the directives field enter the directive in the following format:
   <

>
+rawasm: partition1
+rawasm: partition2
Click on Apply.

Close this window and go to the Clients menu and select Client Setup. 

Select the client with the raw devices. 

In the Saveset field specify:
/dev/partition1
/dev/partition2 

In the Directive field select Raw. 

Click on Apply.

Note:

 

Precautions when using rawasm to back up UNIX raw partitions:

One can specify the rawasm directive to back up raw disk partitions on UNIX.

However, if the raw partition contains data managed by an active database management system (DBMS), ensure that the partition is offline and the database manager is shutdown. For greater flexibility when backing up partitions that contain data managed by a DBMS, use a NetWorker Module application.

Similarly, if rawasm is used to save a partition containing a UNIX file system, the file system must be unmounted or mounted read-only to obtain a consistent backup.

 

Note: Do not specify the rawasm directive to backup or recover raw partitions on Windows. See    Backing up raw partitions on Windows on page 88 provides more information.

Using rawasm to recover a UNIX raw partition

When recovering data, rawasm requires that the file system node for the raw device exist prior to the recovery. This protects against the recovery of a /dev entry and the overwriting of data on a reconfigured disk. You can create the /dev entry, having it refer to a different raw partition, and enforce overwrite if needed. If you create the /dev entry as a symbolic link, the data is recovered to the target of the symbolic link.

 

Recovery of a raw partition must occur on a system configured with the same disk environment and same size partitions as the system that performed the backup:

• If the new partition is smaller than the original partition, the recovery will not complete successfully
• If the new partition is larger than the original partition the estimated size reported upon recovery is not accurate

Note: Only level FULL is supported with rawasm, this is because the original NetWorker traditional backup codes are designed for Filesystem level Backup, and not Block Level backup. 

VADP with NetWorker 7.6.x

I have talked about VADP with hotadd in previous posts.  Last month, I have completed testing on VADP with NetWorker 7.6.2.7.  Turn out I should upgrade to NetWorker 7.6.3.7 because of the following two issue.

NW138379 - VADP : Save does not honor the VSS:*=off flag in Windows 2008
This issue is seen if you proxy host is Windows 2008.

Event 16, source: volsnap, Logged: 1/17/2012 6:25:51 PM
The shadow copies of volume E: were aborted because volume, which contains shadow copy storage for this shadow copy, was force dismounted

Another anonying issue is NW139877 - VADP: nsrvadp_save incorrectly reports status as success for aborted backups

I have not tried NetWorker 7.6.4.  However, I have not encountered those issues after upgrading to NetWorker 7.6.3.7.

Whether virtual or physical proxy, you can consult the following page Nuts and bolts in NetBackup for VMware: Transport methods and TCP ports to decide which transport mode you plan to use.  I don't prefer SAN mode even though it is the fastest option.  Presenting a VMFS lun to a Windows 2008 proxy is not an option in some sites.  If you want to use SAN mode, make sure to control who can login to the proxy.  You don't want Windows signature written to a VMFS lun by mistake. 

NetWorker Block Based Backup failed

There is a new feature introduced in NetWorker 8.x called Block Based Backup.  It is completely rewritten.  So, it is not the same as SnapImage.  It supports Block Based Backup for Windows machine.  For some reason, backup is only successful with Full level backup.  I attempt to start synthetic full backup but it fails.  This is the first client we attempt to use Synthetic full.  We are trying to backup a drive with size of few TB.  Block Based Backup will save the backup time.  We make sure the OS is supported, and BBB is sent to the same media only.  Also, we have weekly full and rest of them are synthetic full.

Eventually, my colleague works with DataDomain and NetWorker support and confirm it is required to enable synthetic backup on DataDomain.  Just login to DataDomain and run the command below.

ddboost option set virtual-synthetics enabled

run the following command to ensure virtual-synthetics is enabled.

ddboost option show

NetWorker EBR backup error "Internal Cycle Error"

Recently, there was a maintenance done in the VM environment.  All the EBR backup failed.  After rebooting the EBR appliance, we still experience intermittent issue with EBR backup.  Random VM backup failed with error " INTERNAL CYCLE ERROR: backstreamdir failed"

Working with support and eventually find out time is not in sync.  From the logs, they find out start time from NetWorker jobs seem to be about 10s off when other logs are checked from VM and EBR side.  We find out from our VM team time sync is turned off in VMtools for some reason and it should be disabled.  Because our NetWorker server and SN is not in domain, it is pointed to different NTP for time sync.  After confirming time sync issue is fixed, we don't experience any more issue.

So, always go from basic to troubleshoot VM backup like the past, are VMtools up-to-date?  ESX servers running the same version in the VM cluster?  Time is in sync?

NetWorker new "save session distribution" option for load balancing

One of the new feature I discover after upgraded to NW 8.1.1.7 is "Save Session distribution".  There is one Networker server and 3 storage nodes to handle the backup.  NW server only handles index, bootstrap and a few Solaris client only.  Majority of the backup are sent to storage nodes.  When a pool is created, we make sure all 3 storage nodes setup a devices to the pool for load balancing.  In the past, you have certain clients go thru sn1 and the other go thru the other sn by setting the storage node affinity of clients.  Here are the few pts I found from the NetWorker training.

Max session

• save sessions are distributed on each sn device's max session attribute (default)
• more likely to concentrate the backup load on few storage nodes. 

Target session

• save sessions are distributed based on each storage nodes device's target sessions attribute
• more likely to spread backup across multiple storage nodes

P.S. Save session distribution is not available for clone or recover operations.

You can set this up from the server's properties.

This configuration can be overridden on a client basis within the client properties.

Lastly, make sure you enter all available and qualified sn in the storage affinity field of the client.  NetWorker will try to load balance the backup among the storage nodes defined in the storage node affinity field.

Migration from NetWorker 7.6 to 8.1.1.8 and dynamic nsrmmd

Finally completed the NW 7.6.5.7 to 8.1.1.7 migration and troubleshooting on Labour Day.  Backup devices are DD880 running DDOS 5.2.x.  Just have a chance to write the post now.  DFA works perfectly fine for the clients and backup window shortens quite a lot.  A new NetWorker 8.1.1.7 server is built and the same client instances are created.  So, no upgrade of existing 7.6.5.7 backup server or mmrecov on the new box.  Clients are moved in a group of 50 - 100 each week.  The old box was virtualized and waited for all saveset expired on the 7.6.5.7 box next yr since we don't set a long retention.

No issue comes up until last group of clients were moved few days before the Labour Day.  After rebooting the server, all NW services started up fine.  I ran a bootstrap backup, and it kept on waiting for media for a long time.  So, I just recycle NetWorker service, and backup was fine.  Few days later, same thing happens again.  Because of tight backup window, I just restarted the service and it was fine again.  So, I opened up a case with support to look for bugs, however, there was nothing similar to my situation.  When it happend again just before the Labour Day, follow routine troubleshooting steps, I discovered the nsrmmd count was not correct during the busiest backup period.  All nsrmmd should be used during the busy backup window but in fact, not all of them were used.  I turned off dynamic nsrmmd as a workaround, and it never happend again.  Once you turn off dynamic nsrmmd, you will see all nsrmmd of the device started.  Looks like NW does not call for more nsrmmd when they are needed.  To turn off dynamic nsrmmd (new feature in 8.x), goto NMC > Devices > Storage Nodes > open up the properties of each SN and de-select "dynamic nsrmmd".  I have not checked patch 8 and 9.  It maybe fixed by now.

When this issue occurs, because there are not enough nsrmmd to serve all the backup, lots of jobs are piled up.  If you run netstat, you will see a lot of TIME_WAIT because of that.

To count the # of nsrmmd, open up the properties of the device.

In this example, we have target sessions start from 1 and max sessions set to 20.  Keep in mind 1 nsrmmd is used for RO for restore session.  The remaining 5 nsrmmd will handle 20 sessions max for backup.  That is 1 nsrmmd handling max of 4 sessions.  I am a bit conservative and still use the ratio suggested in 7.6.1.  I know there are different numbers suggested.  However, I only need to run 1 restore every 2-3 days.  Only need to DR a box once - twice a yr.  So, 1 RO stream is enough.

If you want to achieve the best dedupe ratio for database backup, you can set your device to 1 to 1 for the sessions to nsrmmd ratio.  If you decide that device to support a max of 8 sessions.  Then the max nsrmmd count should be 9 in this case.

savefs: Cannot retrieve the client resources

Recently, colleague has issue backing up a new client.  The following errors keep coming up.

90088:savefs: Cannot find the NSR group resource named 'Windows'.
90069:savefs: Cannot retrieve the client resources.

From my exp, these are DNS related error.  Check the Command Reference Guide. The 2nd possibility is client is moved to another group during backup.  I know this is not the case.  Check DNS and both forward and reverse lookup are fine.

Eventually, find out this new box is setup to replace another box.  The new box is setup with a different hostname but the alias is the same.  Compare the client instance between the new and the decommissioned one. Find out the alias is entered in the decommissioned client (if you attempt to enter alias on the newly created client, it should give you an error because you cannot have 2 different hosts with the same alias).

All we do is remove the alias that are used by the newly built box from the decommissioned client's aliases field.  Then enter it on the newly created client's aliases field.  Now, backup runs fine without any more issue.

(Another commonly see DNS related savefs error is "nothing to save".  Check and make sure aliases field include the other name if there is more than one aliases or one NIC on the host)

NetWorker 8.1.1 EBR VM backup

We deployed NetWorker 8.1 for one of our customer with DD4500 earlier this year.  It works pretty well.  DR and restore are easy.  However, there are a few things to know before you decide to deploy it.

1) EBR and NetWorker server must be resolvable in DNS server (both forward and reverse lookup).  It looks like adding to hosts file is not an option.  Maybe someone want to try hosts file and let me know if they can get it working without DNS.

2) Schedule with override does not seem to work.  For ex, I have schedule with incremental except 1st Fri of the week.  I find out those customized schedules with exception are not available for an "VMWARE action" in the policy.

3) Not enough activity shown in NMC and activity logging in NetWorker.  For ex, when I DR a box, I don't see much info from NMC.  I have to check thru vCenter GUI.

4) If you choose to backup to the internal storage on EBR, you cannot clone them.

5) There is no option to clone on demand.  It is based on policy.  All policy must start as a backup.  So, I cannot just clone them out with nsrclone command.

6) If you plan to backup your VM running SPS, SQL etc, it is not Microsoft application aware backup.  You still need to use NetWorker module for the backup.

7) Need to login to EBR to perform DR.

8) Each proxy can handle only 8 streams.  Deploy more proxy if you have a tighter backup window for your VMs.

Workaround to dump NetWorker database using nsrmmdbasm in NW 8.x

In NetWorker 8.x, due to permission issue, we cannot dump the database out using nsrmmdbasm (assume both nsrexecd and nsrd are run as local system).  To workaround the issue after discuss with support.  There are two workarounds.

1. Start the Networker Remote Exec service and the Backup and Remote Exec service using the same administrator account you are logged into the system.

2. Use the “at” windows command to run nsrmmdbasm under the SYSTEM account.

I choose option 2.  First download a copy of PSexec from Sysinternals and unzip it to a folder. 
Then open Command Prompt as Administrator. 

cd to the folder with PSexec tools. 

Run the following command in command prompt
PSEXEC -i -s -d CMD
This will open up a new command window

Run "whoami /user" to confirm the new command window is run as local system.

Then run the nsrmmdbasm to dump the mm out as a flat file. 
nsrmmdbasm -s "C:\Program Files\EMC NetWorker\nsr\mm\mmvolume6" > mmdb.dump

 

 

 

 

Do not start NetWorker automatically after successful installation

For some reason, if you don't want NetWorker start automatically after successful installation, you can run the command below with setup.exe. 

setup /vSTARTSVC=0

There should be a kb in powerlink if you are interested. 

PS: Please refer to installation guide for more option used with the command line installation.  (7/13/14)

Apply Microsoft fix 2748349 on all Windows backup clients

Other than applying Microsoft patch 940349 on all Windows 2003 server, there is a new patch now for all VSS backup on Windows client - 2748349.  This is required on all VSS backup.  Here are the symption and the prerequisites described in the Microsoft kb 2748349 (please review Microsoft kb for all the details).  Basically, this applies to all backup application using VSS. 

Assume that you have a computer that is running Windows Server 2003, Windows Vista, Windows Server 2008, Windows 7, or Windows Server 2008 R2. You use a backup application that uses the Windows Volume Shadow Copy feature to create snapshots and read files from the snapshots. In this situation, one or more corrupted files are found in the snapshots.

Notes

• The corrupted files can be backed up by using the backup application. The corrupted files can only be noticed during a restore process.
• The backup data of any third-party back up application can be affected. For example, NTBackup, Windows Server Backup, System Center Data Protection Manager (DPM) can be affected. Please contact your third-party backup solution manufacturer to determine whether they use the Windows Volume Shadow Copy feature to create backups.

Prerequisites

To apply this update, you must be running one of the following operation systems:

• Windows Server 2003 Service Pack 2 (SP2)
• Windows Vista Service Pack 2 (SP2)
• Windows Server 2008 Service Pack 2 (SP2)
• Windows 7
• Windows Server 2008 R2 

Note The fix is included in Windows 7 Service Pack 1 (SP1) and Windows Server 2008 R2 SP1.

VSS snapshot is not cleaned up by NetWorker NMM

I have an issue on one of the Windows 2008 client running Sharepoint 2010.  I have to manually stop the backup from NMC.  However, I find out nsrsnap_vss_save is still running on the client.  So I manually stop all Networker service on the client.  However, the VSS snapshot is still there.  I have tried vssadmin to delete shadow but I receive an error the snapshot will not be deleted.  If you run vssadmin list shadows, you can still see the snapshots (borrow output from esg110691)

vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001 Microsoft Corp.

Contents of shadow copy set ID: {5bce2dd1-b18c-49d1-bb1a-51f656d07794}
Contained 5 shadow copies at creation time: 12/20/2009 11:01:01 PM
Shadow Copy ID: {a8019c4f-7dc1-4717-ac1f-affe759a1ea4}
Original Volume: (C:)\\?\Volume{264f5213-8311-11dc-82da-806e6f6e6963}\
Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy1
Originating Machine: w2k3.acme.com
Service Machine: w2k3.acme.com
Provider: 'Microsoft Software Shadow Copy provider 1.0'
Type: ApplicationRollback
Attributes: Persistent, No auto release, Differential, Exposed locally

Shadow Copy ID: {4d5c74fe-7683-4ad8-8cba-cbeaff1adb92}
Original Volume: (D:)\\?\Volume{4efae4e1-834a-11dc-99fe-001a64242e28}\
Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy17
Originating Machine: w2k3.acme.com
Service Machine: w2k3.acme.com
Provider: 'Microsoft Software Shadow Copy provider 1.0'
Type: ApplicationRollback
Attributes: Persistent, No auto release, Differential, Exposed locally

So I contact EMC support and they recommend to try the following three method.

Method 1:
a, if nsrsnap_vss_save.exe is still running in the client while no scheduled backups are running from NetWorker Server, kill that process and it should kill all the subsequent processes.
b, After that please start another NMM scheduled backup, it is supposed to clean up the mess it left behind last time. (This does not happen successfully a number of times).
If method 1 does not work, please kindly try method 2.

Method2：
you can use the following two commands to delete the shadows and the mount points.
vssadmin list shadows
mountvol
These two commands should help you find out what shadows copies are left behind on the system and how these are mounted. Try to unmounts them first using mountvol /D.
/D Removes the volume mount point from the specified directory.
And then use:
vssadmin delete shadows all
That should clean up the rest of the mess manually.
If method 2 does not work, please kindly try method 3.

Method3：
please kindly download VSS SDK from Microsoft:
http://www.microsoft.com/en-us/download/details.aspx?id=23490. and use that instead to get rid of the existing snapshots.

Obviously, I have tried first one.  For the 2nd method, I carefully examined the vssadmin list shadows output and made sure I choose the correct one to umount.  If you pick a data volume by mistake, you can umount a volume in use and cause data unavailable.  However, it still failed.

I didn't go any further since I needed approval from server owner to install the the VSS SDK mentioned in method 3.  I decide to go for google and find out the command to remove snapshot in Windows 2008 is different (my mistake not to tell support the OS version).  I find the instructions from Doug's blog.

I just ran diskshadow then "delete shadows all".  Now, the stuck snapshots are removed.  After that, restarted the backup from NMC and go back to sleep.

Manually uninstall NetWorker in Windows

I try to upgrade NetWorker on one of the Windows client and find out NetWorker is no longer in Add / Remove Program.  If I try to upgrade, it will complain an existing version exist.  Do a bit google search and find the following article in EMC community on how to uninstall NetWorker manually in Windows.

Remove the Upgrade Code from the following registry keys:
HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Installer\Upgrade Codes

HKCR\Installer\UpgradeCodes

Windows Installer uses a GUID (Global Unique Identifier) to uniquely identify products being installed. They call it an Upgrade Code. This is used to identify that NetWorker is installed on this machine.

NetWorker has three upgrade codes (the first for NW 6.0, the second for all other NW versions (except X64), the third for the X64 package):

3EB2C626C9BC4D118811000972CCA7DF
2827A1B508153D114831000A9C877BD1
1D09C7743451200439D99949BD5A9F1E

It may also be necessary to remove the NetWorker services from the registry. They are located in:

-HKLM\SYSTEM\CurrentControlSet\Services
-Delete the following keys: nsrd, nsrexecd, nsrpm, hagentd, lgtolmd
-For NMC, the key to look for is: gstd
-After removing the services, reboot the machine.

Also run the below commands on the command prompt before you reboot the machine

sc delete nsrexecd
sc delete nsrd

All installed products have an entry in the following registry key:
-HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall

-Control Panel Add/Remove programs gets its information from this key.
-If an entry under this key is deleted, it will not be displayed in Add/Remove Programs.

The above steps work for me.  The version of NetWorker on my box is 7.6.1.  You may need to delete the following key.


HKLM\Software\Legato\NetWorker

To manually remove the old binary.  Be very careful if it is a NetWorker server.
If it is a NetWorker server running on Windows, NEVER delete res and mm folders (that's where the NetWorker config and media database located).  I normally keep the logs folder too.  I believe the dedupe folder contain the cache info for Avamar.  

Again, backup your registry keys before making changes to your registry.

Open Solaris and Debian on NetWorker

Debian, Ubuntu and Open Solaris are not in NetWorker support matrix.  So don't expect to get any support from EMC.

For Debian, so far, I don't encounter issue by following the steps in the uWaterloo website to install NetWorker client software.  I had issue with IPV6 with NetWorker in Debian.  Backup is fine with IPV6 turned off.

I obtain the binary from PowerLink.

1. Install alien to convert rpm packages 
2. apt-get install alien
3. Convert to debian packages and install
4. alien --to-deb -i lgtoman-7.6.4-1.x86_64.rpm
5. alien --to-deb -i --scripts lgtoclnt-7.6.4-1.x86_64.rpm (ignore scripts warning)
6. cd /etc/init.d
7. Get script networker: Legato Networker Startup Script
8. chmod 755 networker
9. update-rc.d networker defaults
10. Update /nsr/res/servers adding backup.cs.uwaterloo.ca (/nsr directory isn't created until after networker is started the first time, even then the servers file isn't created)
11. ./networker start

From uWaterloo website, there is a reminder for Ubuntu.
Ubuntu 10.10 now has a recover command that can undelete files and conflicts with the Legato install

I find out Open Solaris will not work in NetWorker from a blog.  I have never tested it but it recommends 7.4.5 seems to work.

========================================================================
NetWorker 8 support Debian and Ubuntu.  Open Solaris is not supported though.  Check EMC Software Compatibility Guide for latest information.

nsrwatch on NetWorker 7.6.2 in Windows platform

Just upgrade to NetWorker 7.6.2.  nsrwatch is available for Unix and Linux in the past.  Now, it is added to the Windows platform.  Since the Management Console GUI is slow, this really saves a lot of hassle.

10 GbE performance troubleshooting 1

On a Windows 2003 server with 1 GbE NIC and DSN writing to DataDomain device, I can see about 140 MB/s (dedup happens on the DSN and NIC utilization is approx 15%).

Now, with 2 Windows 2008 R2 servers setup with 10GbE, I copy a file from one Windows 2008 R2 to another one.  It at most utilizes 12% of the 10GbE.  If I add write another file at the same time, I see the utilizes 20% of the 10 GbE.  I follow some of the suggestions by Cisco to tweat the OS (only thing I have not done is Jumbo frame).  However, I don't see much improvement.

After doing more research, it looks like it is an OS limitation.  See kb article from HP site.

"There was still perceived TCP performance issue, but it turned out to be a matter of limitations in performance per thread in Windows Server 2003. For instance, if copying only one file from one server hosting a NC522SFP to another using a NC522SFP, only a small fraction of the theoretical 10-Gigabit performance was achieved. However, if multiple sessions were run simultaneously, similar performance gains were seen as with UDP. In other words, the bottleneck was not the NIC."

Hopefully, I will have more time to run test and determine the limitation in the summer.  Not sure if Linux / Unix will do a better job.

DataDomain support with NetWorker 7.6.2 and 7.6.3

With NW 7.6.2 boost devices, max session supported / devices are increased.  Max default session / device will be 10 instead of 4 in NW 7.6.1.  Dedup ratio will not be affected since it is SN side dedup for boost device.  I remember if AFTD is used, set the max session to 1 per device in 7.6. 

With NW 7.6.3, multiplexing is supported for VTL in DataDomain. 

Keep in mind as mentioned in the older article, NW 7.6.3 does not support DDOS 4.9.  If you plan to upgrade from NetWorker 7.6.1 to 7.6.2, make sure you go through the DataDomain Integration guide for NetWorker 7.6.2.  My plan is to stage off the saveset from old DD devices after migration to the new one generated in NW 7.6.2.  This seems to be the simplest option. 

Once migration completed, a case will be open with DataDomain to remove the old unused LSU.  You cannot delete old LSU from NetWorker / DataDomain GUI.  Do not delete LSU that contain data. 

Never mixing NDMP and non-NDMP backup in the same media for NetWorker

The following are the limitation of NDMP backup with NetWorker.  You can find these in admin guide but most ppl tend to skip them.  It will be a nightmare when you find out data cannot be recovered.  Some of these limitation possibly apply to other backup solution as well.

For pure NDMP backup (backup to NDMP devices.  that is, FC tape drive presented to NAS), make sure you have non-NDMP backup sent to different pool including index and bootstrap.  The same applied to cloning.  So media written by NDMP devices should not be accessed by non-NDMP devies.  It is not possible to save data from any NAS filer to an tape device using standard NDMP and then write non-NDMP data to the same tape volume.  You will have issue during recovery.

NDMP and non-NDMP savesets can be saved on the same volume only if the NDMP backups to that volume were written using NDMP-DSA (NAS data backup to device presented to Storage Node / NetWorker server).

There is an old kb article from Powerlink to explain it.
"When non-NDMP backups are written to tape, the backup is written directly to the tape by NetWorker in it's own proprietary format. When NDMP and non-NDMP data is written to the same tape, the file marks get out of order and as a result the restore cannot position the tape correctly to find the data image.
Fix: Ensure that NDMP data be backed up to it's own separate pool to prevent NDMP recovery and scanner problems."

Keep in mind when migrating your NAS.  If you plan to migrate to different vendor, for ex, NetApp to BlueArc, make sure you no longer need the backup on the old NAS (in this ex, NetApp) or you keep your old NAS somewhere in case recovery required.  NetApp and BlueArc are on different OS.  That's why you cannot recover NDMP backup by NetApp to BlueArc.

Also, make sure you have a copy of the index saveset.  scanner -i does not work for NDMP backup.

For full details of limitation and other requirements of NetWorker NDMP backup, please consult EMC support or refer to EMC documentation.