Manually uninstall NetWorker in Windows

I try to upgrade NetWorker on one of the Windows client and find out NetWorker is no longer in Add / Remove Program.  If I try to upgrade, it will complain an existing version exist.  Do a bit google search and find the following article in EMC community on how to uninstall NetWorker manually in Windows.

Remove the Upgrade Code from the following registry keys:
HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Installer\Upgrade Codes

HKCR\Installer\UpgradeCodes

Windows Installer uses a GUID (Global Unique Identifier) to uniquely identify products being installed. They call it an Upgrade Code. This is used to identify that NetWorker is installed on this machine.

NetWorker has three upgrade codes (the first for NW 6.0, the second for all other NW versions (except X64), the third for the X64 package):

3EB2C626C9BC4D118811000972CCA7DF
2827A1B508153D114831000A9C877BD1
1D09C7743451200439D99949BD5A9F1E

It may also be necessary to remove the NetWorker services from the registry. They are located in:

-HKLM\SYSTEM\CurrentControlSet\Services
-Delete the following keys: nsrd, nsrexecd, nsrpm, hagentd, lgtolmd
-For NMC, the key to look for is: gstd
-After removing the services, reboot the machine.

Also run the below commands on the command prompt before you reboot the machine

sc delete nsrexecd
sc delete nsrd

All installed products have an entry in the following registry key:
-HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall

-Control Panel Add/Remove programs gets its information from this key.
-If an entry under this key is deleted, it will not be displayed in Add/Remove Programs.

The above steps work for me.  The version of NetWorker on my box is 7.6.1.  You may need to delete the following key.


HKLM\Software\Legato\NetWorker

To manually remove the old binary.  Be very careful if it is a NetWorker server.
If it is a NetWorker server running on Windows, NEVER delete res and mm folders (that's where the NetWorker config and media database located).  I normally keep the logs folder too.  I believe the dedupe folder contain the cache info for Avamar.  

Again, backup your registry keys before making changes to your registry.

HP Advisory on 10G NC522SFP with connectivity issue

Recently, one of the servers with 10G NIC starts to show the following error message. 

DEVICE: HP NC522SFP Dual Port 10GbE Server Adapter #2
PROBLEM: Tx path is hung. The device is being reset.
ACTION: Adapter recovers from this error automatically.

Search from the web and see this HP advisory.  When I attempt to upgrade the firmware, I get the following error. 

"Dependencies Failed -- Driver and firmware set are not allowed for update. Please install driver version 4.3.11.407 prior to performing firmware upgrade on devices with firmware version 4.0.550 or earlier installed. See nxflash.log for further details"
Upgrade to the latest driver and update the firmware.  Failed!

Upgrade to driver version 4.3.11.407 and then update the firmware.  Failed again!

Call support and they have me redo the same thing again.  Testing different firmware version and going nowhere.  After 3 hrs of troubleshooting, HP tells me it is a bad NIC.  I refuse to get a replacement NIC since the NIC is functioning fine.  So, I arrange another downtime and use my a spare 10G NIC.  I still have the same issue.  Eventually, we put the driver back to 4.3.11.407.  After that, they escalate to next level.  Suggestion is to use Local Admin account but it is still going nowhere (it is true that even domain admin account will return the same error.  You need to login as local admin to complete the firmware upgrade).

Check back on HP site and see firmware version 1.3.9.7 is available.  Run the installation and it passes the dependencies check.  So, I know it must be dependencies issue on firmware not driver. 

The solution is to stay at driver 4.3.11.407 and update firmware to 1.3.9.0.  Reboot.  Then update firwmare to 1.3.9.6.  Reboot.  Run the driver update and issue is now resolved.  If your NC522SFP firmware is too old, upgrade to firmware 1.3.9.0 first. 

Bodhi Linux 2.1 on an old AMD Sempron

Recently read about lightweight Linux and find out about Bodhi Linux 2.1.  So, I decide to install Bodhi Linux 2.1 on it.  The machine spec is 512 MB of memory and AMD Sempron socket A 2500+ CPU.  Install is very straight forward.  The following software are installed from Add Software option.

Firefox 16.0
Java JDK 7 u5
LibreOffice 3.x
Adobe Reader
Printing for my Brother HL-3070CW

This is for basic office application and internet banking.  I am very satisfied with the performance.  At least, boot time and response time is much better than CentOS 3.  Forget about XP, it barely runs on this box.

Here are some of the issue I encounter.

I cannot read Chinese in Firefox.  So, I do some research and it requires the following packages (see reference).  Use Synaptic Package Manager to install them.

• ttf-arphic-bkai00mp
• ttf-arphic-bsmi00lp
• ttf-arphic-gbsn00lp
• ttf-arphic-gkai00mp
• ttf-arphic-ukai
• ttf-arphic-uming
• 
  

For Flash, I see people complaining about viewing youtube (see reference).  I follow the instructions to download and install mint-flashplugin-11 package.  It works flawlessly.

After installed printing package (basically CUPS), then I select the printer from the list.  For some reason, I don't see an option to select Mono or Color printing.  So, download the squeeze package from Brother website and follow the instructions to install lpr package then the cupswrapper driver.

Go back to http://localhost:631/printers to modify the printer to select HL-3070CW again.  This time, I do see the Color/Mono option under Printer Options of Printer properties.

I try to download VMWare WS 7 and test it on this box.  However, it won't compile.  Someone point out similar issue and resolve by installing WS 8.  However, WS 8 requires 64 bit processor even for 32 bit version (see this).  It affects my plan to test Bodhi Linux with VMware WS 7.1.6 on my old Sempron since I plan to replace CentOS on my Lenovo T400 laptop with Bodhi Linux 2.1.

SCSI reservation conflict error in ESX

Basically, if you encounter such an error in your ESX environment,

WARNING: SCSI: 5532: Failing I/O due to too many reservation conflicts

WARNING: SCSI: 5628: status SCSI reservation conflict, rstatus 0xc0de01 for vmhba1:0:7. residual R 919, CR 0, ER 3
WARNING: J3: 1970: Error committing txn to slot 0: SCSI reservation conflict

you should follow VMware kb 1005009. 

On the storage side, make sure you follow best practise guide.  For example, array fan in limitation.  For Symmetrix, make sure the front end port is dedicated for ESX connection only and do not mix with other host running other OS. 

Error Codes list for Microsoft technologies

When we troubleshoot backup issue, especially system state backup, backup program normally return a Microsoft error code.  For example, in NetWorker, you may get an error like

"ASR System Files - AsrCreateStateFile failed - 0x80080005"

(By the way, if you get this error for system state backup, most likely a reboot will fix it)

To find out what it means, there is a Symantec kb "Error Codes list for Microsoft technologies" listing the MS error codes.

Over the weekend, I got an alert about System State backup on Windows 2000. Error is 0x00000070.  It complains there is not enough space.  Confirm by rdp to the box and only 3 MB available on the system drive.  An old Windows 2000 box waiting to be retired.

Open Solaris and Debian on NetWorker

Debian, Ubuntu and Open Solaris are not in NetWorker support matrix.  So don't expect to get any support from EMC.

For Debian, so far, I don't encounter issue by following the steps in the uWaterloo website to install NetWorker client software.  I had issue with IPV6 with NetWorker in Debian.  Backup is fine with IPV6 turned off.

I obtain the binary from PowerLink.

1. Install alien to convert rpm packages 
2. apt-get install alien
3. Convert to debian packages and install
4. alien --to-deb -i lgtoman-7.6.4-1.x86_64.rpm
5. alien --to-deb -i --scripts lgtoclnt-7.6.4-1.x86_64.rpm (ignore scripts warning)
6. cd /etc/init.d
7. Get script networker: Legato Networker Startup Script
8. chmod 755 networker
9. update-rc.d networker defaults
10. Update /nsr/res/servers adding backup.cs.uwaterloo.ca (/nsr directory isn't created until after networker is started the first time, even then the servers file isn't created)
11. ./networker start

From uWaterloo website, there is a reminder for Ubuntu.
Ubuntu 10.10 now has a recover command that can undelete files and conflicts with the Legato install

I find out Open Solaris will not work in NetWorker from a blog.  I have never tested it but it recommends 7.4.5 seems to work.

========================================================================
NetWorker 8 support Debian and Ubuntu.  Open Solaris is not supported though.  Check EMC Software Compatibility Guide for latest information.

Hotadd limitation

Other than the 1TB vmdk limit with early version of VDDK (fixed in VDDK 1.1.1), the following are the limit and suggestion of HotAdd (similar info can be found in vendor support documentation).

1)Hot-Add only works on SCSI disks, not IDE disks
2)Disable automount on proxy host
3)For block size mismatch issue, the VMFS containing the Target VM and the VMFS containing the VADP proxy must use the same or larger block size. For example, if you back up virtual disk on a datastore with 4Mb blocks, the proxy (vRanger) must also be on a datastore with 4Mb blocks or above. Its strongly recomended to install proxy on VM which is residing on 8Mb.

Other consideration not only applied to hotadd but also other transport mode.
1) change the workingDir to a datastore with enough block size if there are more than 1 vmdk for a VM and they are on different datastore with different block size.  For ex, if C drive is on datastore with 1 MB block size, and D drive is on a datastore with 2 MB block size.  workingDir should be on datastore with 2 MB block size.  See VM KB: 1012384

P.S. Do not use hotadd if there is dynamic disk in the VM.  It is just a headache.  See VM KB: 2006279

VADP application quiescing

Recently, I am helping out a friend on VADP deployment.  He has question about VADP quiescing.  I pointed him to an article in Backup Central.  To make sure application quiescing is supported for Windows 2008 R2, make sure ESX 4.1 U1 is deployed (documented in VMware KB: 1031298).  However, if Windows 2008 or Windows 2008 R2 VM deployed under ESX 4.0, make sure KB: 1028881 is followed.

For Exchange and SQL, I still suggest use of Exchange or SQL agent from backup software to backup them up.  When restore is required, for ex, Exchange, you can restore folder in a mailbox easily.

nsrwatch on NetWorker 7.6.2 in Windows platform

Just upgrade to NetWorker 7.6.2.  nsrwatch is available for Unix and Linux in the past.  Now, it is added to the Windows platform.  Since the Management Console GUI is slow, this really saves a lot of hassle.

10 GbE performance troubleshooting 1

On a Windows 2003 server with 1 GbE NIC and DSN writing to DataDomain device, I can see about 140 MB/s (dedup happens on the DSN and NIC utilization is approx 15%).

Now, with 2 Windows 2008 R2 servers setup with 10GbE, I copy a file from one Windows 2008 R2 to another one.  It at most utilizes 12% of the 10GbE.  If I add write another file at the same time, I see the utilizes 20% of the 10 GbE.  I follow some of the suggestions by Cisco to tweat the OS (only thing I have not done is Jumbo frame).  However, I don't see much improvement.

After doing more research, it looks like it is an OS limitation.  See kb article from HP site.

"There was still perceived TCP performance issue, but it turned out to be a matter of limitations in performance per thread in Windows Server 2003. For instance, if copying only one file from one server hosting a NC522SFP to another using a NC522SFP, only a small fraction of the theoretical 10-Gigabit performance was achieved. However, if multiple sessions were run simultaneously, similar performance gains were seen as with UDP. In other words, the bottleneck was not the NIC."

Hopefully, I will have more time to run test and determine the limitation in the summer.  Not sure if Linux / Unix will do a better job.