Thursday, September 23, 2021

NetWorker NDMP backup for Isilon failed after upgrade from 19.4.0.1 to 19.4.0.3

All NDMP backup for Isilon failed after upgraded from 19.4.0.1 to 19.4.0.3. There is no change and firmware update on Isilon side. From NetWorker log, handshake is fine, and backup seems to start. 

42617:nsrndmp_save: NDMP Service Log: 
Filetransfer: Transferred 1271296 bytes in 0.125 seconds throughput of 9969.966 KB/s 
Total 1271296 bytes in this backup stream 

However, the log shows error below. When I monitor from Isilon, I don't see read activity after first few min. 

42923:nsrndmp_save: NDMP Service Error: Failed to write data at offset 220672 
42738:nsrndmp_save: Data server halted: Error during the backup. 

The error seems to point to write issue on DataDomain side. However, all other backup is fine writing to the same DD. The workaround is to disable client direct and use the old trick to use -P option to specify the storage node for NDMP backup command in NetWorker.

==================================================================== 
Forget to mention: we have 2 sites with the same upgrade. The other site works fine. Only difference between them is multi-home. The site with issue is multihome. The site that works fine after upgrade setup with dedicated backup network.

To check from Isilon side, look for NDMP logs in /ifs/.ifsvar/modules/ndmp/trade/node<ID>-ndmpd.trace

No comments: