Sunday, December 2, 2012

VSS snapshot is not cleaned up by NetWorker NMM

I have an issue on one of the Windows 2008 client running Sharepoint 2010.  I have to manually stop the backup from NMC.  However, I find out nsrsnap_vss_save is still running on the client.  So I manually stop all Networker service on the client.  However, the VSS snapshot is still there.  I have tried vssadmin to delete shadow but I receive an error the snapshot will not be deleted.  If you run vssadmin list shadows, you can still see the snapshots (borrow output from esg110691)

vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001 Microsoft Corp.

Contents of shadow copy set ID: {5bce2dd1-b18c-49d1-bb1a-51f656d07794}
Contained 5 shadow copies at creation time: 12/20/2009 11:01:01 PM
Shadow Copy ID: {a8019c4f-7dc1-4717-ac1f-affe759a1ea4}
Original Volume: (C:)\\?\Volume{264f5213-8311-11dc-82da-806e6f6e6963}\
Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy1
Originating Machine: w2k3.acme.com
Service Machine: w2k3.acme.com
Provider: 'Microsoft Software Shadow Copy provider 1.0'
Type: ApplicationRollback
Attributes: Persistent, No auto release, Differential, Exposed locally

Shadow Copy ID: {4d5c74fe-7683-4ad8-8cba-cbeaff1adb92}
Original Volume: (D:)\\?\Volume{4efae4e1-834a-11dc-99fe-001a64242e28}\
Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy17
Originating Machine: w2k3.acme.com
Service Machine: w2k3.acme.com
Provider: 'Microsoft Software Shadow Copy provider 1.0'
Type: ApplicationRollback
Attributes: Persistent, No auto release, Differential, Exposed locally


So I contact EMC support and they recommend to try the following three method.

Method 1:
a, if nsrsnap_vss_save.exe is still running in the client while no scheduled backups are running from NetWorker Server, kill that process and it should kill all the subsequent processes.
b, After that please start another NMM scheduled backup, it is supposed to clean up the mess it left behind last time. (This does not happen successfully a number of times).
If method 1 does not work, please kindly try method 2.

Method2:
you can use the following two commands to delete the shadows and the mount points.
vssadmin list shadows
mountvol
These two commands should help you find out what shadows copies are left behind on the system and how these are mounted. Try to unmounts them first using mountvol /D.
/D Removes the volume mount point from the specified directory.
And then use:
vssadmin delete shadows all
That should clean up the rest of the mess manually.
If method 2 does not work, please kindly try method 3.

Method3:
please kindly download VSS SDK from Microsoft:
http://www.microsoft.com/en-us/download/details.aspx?id=23490. and use that instead to get rid of the existing snapshots.

Obviously, I have tried first one.  For the 2nd method, I carefully examined the vssadmin list shadows output and made sure I choose the correct one to umount.  If you pick a data volume by mistake, you can umount a volume in use and cause data unavailable.  However, it still failed.

I didn't go any further since I needed approval from server owner to install the the VSS SDK mentioned in method 3.  I decide to go for google and find out the command to remove snapshot in Windows 2008 is different (my mistake not to tell support the OS version).  I find the instructions from Doug's blog.

I just ran diskshadow then "delete shadows all".  Now, the stuck snapshots are removed.  After that, restarted the backup from NMC and go back to sleep.