10 GbE performance troubleshooting 1

On a Windows 2003 server with 1 GbE NIC and DSN writing to DataDomain device, I can see about 140 MB/s (dedup happens on the DSN and NIC utilization is approx 15%).

Now, with 2 Windows 2008 R2 servers setup with 10GbE, I copy a file from one Windows 2008 R2 to another one.  It at most utilizes 12% of the 10GbE.  If I add write another file at the same time, I see the utilizes 20% of the 10 GbE.  I follow some of the suggestions by Cisco to tweat the OS (only thing I have not done is Jumbo frame).  However, I don't see much improvement.

After doing more research, it looks like it is an OS limitation.  See kb article from HP site.

"There was still perceived TCP performance issue, but it turned out to be a matter of limitations in performance per thread in Windows Server 2003. For instance, if copying only one file from one server hosting a NC522SFP to another using a NC522SFP, only a small fraction of the theoretical 10-Gigabit performance was achieved. However, if multiple sessions were run simultaneously, similar performance gains were seen as with UDP. In other words, the bottleneck was not the NIC."

Hopefully, I will have more time to run test and determine the limitation in the summer.  Not sure if Linux / Unix will do a better job.

DataDomain support with NetWorker 7.6.2 and 7.6.3

With NW 7.6.2 boost devices, max session supported / devices are increased.  Max default session / device will be 10 instead of 4 in NW 7.6.1.  Dedup ratio will not be affected since it is SN side dedup for boost device.  I remember if AFTD is used, set the max session to 1 per device in 7.6. 

With NW 7.6.3, multiplexing is supported for VTL in DataDomain. 

Keep in mind as mentioned in the older article, NW 7.6.3 does not support DDOS 4.9.  If you plan to upgrade from NetWorker 7.6.1 to 7.6.2, make sure you go through the DataDomain Integration guide for NetWorker 7.6.2.  My plan is to stage off the saveset from old DD devices after migration to the new one generated in NW 7.6.2.  This seems to be the simplest option. 

Once migration completed, a case will be open with DataDomain to remove the old unused LSU.  You cannot delete old LSU from NetWorker / DataDomain GUI.  Do not delete LSU that contain data. 

Never mixing NDMP and non-NDMP backup in the same media for NetWorker

The following are the limitation of NDMP backup with NetWorker.  You can find these in admin guide but most ppl tend to skip them.  It will be a nightmare when you find out data cannot be recovered.  Some of these limitation possibly apply to other backup solution as well.

For pure NDMP backup (backup to NDMP devices.  that is, FC tape drive presented to NAS), make sure you have non-NDMP backup sent to different pool including index and bootstrap.  The same applied to cloning.  So media written by NDMP devices should not be accessed by non-NDMP devies.  It is not possible to save data from any NAS filer to an tape device using standard NDMP and then write non-NDMP data to the same tape volume.  You will have issue during recovery.

NDMP and non-NDMP savesets can be saved on the same volume only if the NDMP backups to that volume were written using NDMP-DSA (NAS data backup to device presented to Storage Node / NetWorker server).

There is an old kb article from Powerlink to explain it.
"When non-NDMP backups are written to tape, the backup is written directly to the tape by NetWorker in it's own proprietary format. When NDMP and non-NDMP data is written to the same tape, the file marks get out of order and as a result the restore cannot position the tape correctly to find the data image.
Fix: Ensure that NDMP data be backed up to it's own separate pool to prevent NDMP recovery and scanner problems."

Keep in mind when migrating your NAS.  If you plan to migrate to different vendor, for ex, NetApp to BlueArc, make sure you no longer need the backup on the old NAS (in this ex, NetApp) or you keep your old NAS somewhere in case recovery required.  NetApp and BlueArc are on different OS.  That's why you cannot recover NDMP backup by NetApp to BlueArc.

Also, make sure you have a copy of the index saveset.  scanner -i does not work for NDMP backup.

For full details of limitation and other requirements of NetWorker NDMP backup, please consult EMC support or refer to EMC documentation.

NetWorker NDMP DSA performance

We try to backup Celerra VG2 using NetWorker NDMP.  From existing hardware, we decide to use NDMP DSA since we don't have license for VTL on DataDomain.  We backup 4 streams together and the total throughput is 30 - 40 MB/s.

(To find out backup performance on Celerra, logon to control station and run the following command: assuming you are backing up filesystem on Data Mover server_2)
server_pax server_2 -s -v

We contact support and they suggest us to use the -P option for the backup command where sn is the storage node you want the backup data sent to.

nsrndmp_save -T dump -M -P sn

We also list only the sn in the storage node field on the client properties of the Celerra in NetWorker.

After making the changes, we see performance improve to 100 MB/s.

Keep in mind, if you are running pure NDMP backup (that is, having a FC tape drive connected to Celerra Data Mover), the backup command will be

nsrndmp_save -T dump

i.e. For most NAS, dump is the only option supported.  Please refer to NetWorker and Celerra documentation for more info.
-M option is for DSA backup.
For pure NDMP, do not add -M in the backup command.

Data Domain ifgroup part III

We try to config ifgroup with NetWorker 7.6.1, and it fails to work as expected.  For ex, if I setup ifgroup with 2 dual 10GbE NICs, and unplug a cable.  Sometimes, I don't see the backup session failovers to the other 3 working NIC.    If I set up failover on the NICs of DataDomain, I don't have any issue as mentioned in my earlier post.  Contact support and the suggestion is to upgrade to NetWorker to 7.6.2 and the DDOS to 5.0.  I am happy with NetWorker 7.6.1 and DDOS 4.9.x for now.  Will decide when to upgrade in the future.  Currently NIC failover on DataDomain is good enough!

VADP with hotadd configuration

Helper VM is not required for VADP with ESX 4.0 or above if hotadd mode is selected.  Make sure you do have the necessary license for hotadd.  If VCB is still used in the environment, you can follow Commvault's page "Create a Helper Virtual Machine" on how to configure VCB helper VM.

The following is the properties of the proxy VM before snapshot being mounted. 

After snapshot is created, it will be mounted on the proxy host (which is a Virtual machine).  See screenshot below. 

I have backup 2 VM with one disk on each VM at the same time.  That's why you see two extra virtual disk on the proxy host.  Now, you probably understand why you need Advanced version for ESX 4.1 which include the hotadd feature.  Otherwise, you can only use nbd or san transport mode to backup VM. 

Of course, there is limitation on hotadd such as max vmdk size of 1 TB.

Actually, the 1 TB limitation is no longer valid for VDDK 1.1.1 (http://www.vmware.com/support/developer/vddk/VDDK-1.1.1-Relnotes.html) 

If NBD is used, there is a 1 TB limitation.  

DataDomain link aggregation 10GbE part II

After reviewing DDOS 4.9 Initial Config Guide, link aggregation on 10GbE is not supported.  DDOS 4.9 only supports link aggregation on 10GbE for 1 GbE.  This is confirmed by support as well.

"The 10 Gb-to-10 Gb interface does not support link aggregation. Only 1 GbE ports are supported."

Only failover will be supported on 10GbE for DDOS 4.9 and 4.8.  However, do consider the following before configuring failover.

• 10 GbE copper-to-10 GbE copper ports across Intel NICs can be
combined for failover (but not for link aggregation).
• 10 GbE optical-to-10 GbE optical ports can be combined for
failover (but not for link aggregation).
• 10 GbE failover across Intel NICs is supported.

VADP / VCB backup failed with hotadd mode

I am testing the VADP backup and keep getting the following error.

"Unable to open a disk of the virtual machine"

Monitor the VC and do see snapshot created.  Run the same backup with nbd and it works fine.  Double check and find out ESX license in on Standard Edition with version 4.1.  

Remember this is a license issue and confirm from VM site VDDK 5.0 release note.

"Licensing. In vSphere 5.0, the SCSI HotAdd feature is enabled only for vSphere editions Enterprise and higher, which have Hot Add licensing enabled. No separate Hot Add license is available for purchase as an add-on. In vSphere 4.1, Hot Add capability was also allowed in Advanced edition. Therefore, customers with vSphere Essentials or Standard edition who use backup products (including VMware Data Recovery) are not able to perform proxy-based backup, which relies on SCSI HotAdd. Those customers must use alternate transport modes."

The most important benefit of using hotadd mode to me is to avoid presenting a VMFS lun to Windows proxy server.

P.S. HotAdd is not supported with VDDK 1.1.1 when proxy server is on ESX 4.1 even correct license is purchased.  See reference link VDDK-1.2.1 release notes

Error labeling new boost device on DataDomain

NetWorker 7.6.3 is out and I try to install the new code on my test VADP proxy.  When I attempt to label a new media for my new boost device,  I get the following error message.

nsrd rd=networkerSN:datadomain_DD0304 mount operation failed: Connecting to 'datadomain' failed ([5028] rpc connection failure).

From experience, this is DNS and connectivity related issue.  Check DNS records and even add all the entries to host file.  It still return the same error.  Finally, uninstall NetWorker 7.6.3 and install NetWorker 7.6.2 back to the test box, everything works fine.

Check release notes and find out NetWorker 7.6.3 requires DDOS running at 5.0 or higher if you are using DDBoost.

Only DDOS versions 5.0.x and 5.1.x are supported with DD Boost 2.4

Currently, DDOS is running at 4.9.2.x and I don't plan to upgrade to DDOS 5.0 any time soon.  Only other option is to use AFTD.

Expanding boot partition of Windows 2008 server

Originally, I have two partition on only one disk.  Finally, I get my new disk.  So, I use Ghost to copy the second partition D drive to the new disk.  To free up space, I remove the old partition D drive on the 1st disk in Disk Management.  Then right click on C drive and choose Extend Volume.  Follow the wizard and now C drive is expanded to occupy all the space on the first disk without reboot.

There is a Shrink Volume option.  Not sure how reliable it is though.