Upgrade to SupportAssist in Isilon

After upgrade to OneFS 9.7.1.4, there is a feature enable to use Support Assist for dial in and alerting EMC.  It will be mandatory in the future release of OneFS.  

We have latest version of SCG to monitor the Isilon.  So, we decide to enable Support Assist via Gateway topology.  

Links below give you the info about Support Assist for Isilon and how to enable Support Assist in Isilon.

https://infohub.delltechnologies.com/sv-se/p/onefs-supportassist-provisioning-part-1/

https://infohub.delltechnologies.com/sv-se/p/onefs-supportassist-provisioning-part-2/

For CLI to migrate to Support Assist, see EMC Article Number: 000225888.  

https://www.dell.com/support/kbdoc/en-uk/000225888/powerscale-how-to-migrate-from-secure-remote-support-srs-to-supportassist

Since there is existing SCG monitoring Isilon, the only thing I have to add is a firewall rule to allow traffic from all Isilon nodes to both SCG GW port 9443. 

It will take up to 24 hours for the provisioning to complete.  See EMC KB Article Number: 000232371

https://www.dell.com/support/kbdoc/en-us/000232371/powerscale-support-assist-provision-gets-stuck-in-incoming

======================================================================

Support Assist is not sending alerts back to EMC.  

Check from SCG and see connectivity is bad but CLI is ok.  Have Dell dial in without issue.  After going back and forth, it is actually a bug.  The connection will stay up for few days to few weeks then it will not dial back to EMC.  See EMC KB Article Number: 000348244

PowerScale: SupportAssist Disconnects SCG and CIQ - APEX AIOps Due to Crispies Lock on Node

We decide not to upgrade and stay in 9.7.1.4.  So, support has to follow the kb to apply the workaround.  

The other bug is email alert once a month to remind you the cluster is not connected back to EMC via eSRS like below.  After migrating to SupportAssist, it looks like eSRS is now shutdown in Isilon.  Check with support and this can be ignored since Isilon is now monitored by SupportAssist.  Wonder if this will be fixed in future release.  

"This cluster is not connected back to EMC via ESRS.

Configuring connectivity to EMC via ESRS is a secure method to improve the overall reliability of your cluster and reduce the time it takes to resolve any issues the cluster may develop."

NDFC upgrade / deploy part 3 - deploy new NDFC one node setup in ESX

 As mentioned in previous post, we choose 1 node setup with App OVA with 16vCPU and 64GB memory.  It will be run in ESX cluster.  

Pls review the link below 

Deploying Nexus Dashboard Using VMware vCenter

Cisco Nexus Dashboard Fabric Controller 12 

For SAN, you still need the following ready before deployment.

1) 2 NICs: one for management IP and one for data IP and both can be in the same subnet.  

2) DNS and NTP are required

3) Cluster name and VM name should be different but it is a one node setup.  So, it does not really matter.

4) Pool IP: 2 are required in our setup (2 IPs for data network)  Ask for an additional IP if you plan to use  SAN Insight).  All pool IP should be whitelisted for SMTP traffic in additional to Management and Data IP.  

5) Internal subnets settings (these subnets are used internally and will not leave cluster.  Just in case, confirm the subnets are not in use and subnet mask has to be 255.255.0.0)

App subnet and Service subnet

6) May want extra space for HDD depends on the number of switches to manage.

The steps are very straight forward.  However, depending on the environment, do not specify VLAN in step 18 for data network.  Otherwise, the data network will not be able to communicate with the switches even they are in the same subnet.  In our environment, VLAN info is already present at the switch and ESX level.  

For initial setup, you can follow the document below 

Initial setup

Feature Management will be set to SAN Controller with Performance Monitoring and SAN Web Device Manager for my environment.  

Server settings are self explanatory.  For SMTP, make sure all pool IPs are added to SMTP allowed list.  

If you want to confirm SMTP alert, go to Event -> Event setup -> Forwarding -> Add Rule (same as DCNM).  Multiple recipients allowed and event email will be sent thru pool IP.  

Below link provided detail steps on how to add event email.

How to Configure Event Forwarding using Cisco Nexus Dashboard Fabric Controller - Cisco

I include a blog post below for bugs we faced for NDFC appliance.  See 

NDFC bugs since deployed

NDFC upgrade / deploy part 2 - sizing and compatibility

At the time we were planning, we decided to use ND 3.2.1i (which comes with NDFC 12.2.2).  That would work with all our existing linecard, switches and firmware.  I don't need to worry about existing firmware 8.4(2e) was not supported.  Please use links below for compatibility check.  

For software / hardware compatibility, pls check NDFC Software and Hardware Compatibility Matrix.

To confirm the NDFC and ND compatibility, pls check Cisco Nexus Dashboard and Services Compatibility Matrix    

Next is sizing.  For sizing, pls check Cisco Nexus Dashboard Capacity Planning.  We decide to use 1 App node config since it is a small environment.  (Note: if one node config is decided, it will not support adding any more node in the future).  App OVA requires 16 vCPU and 64GB memory.  

Other requirement: pls go through documents below

Cisco Nexus Dashboard and Services Deployment and Upgrade Guide, Release 3.2.x - Prerequisites: Nexus Dashboard [Cisco Nexus Dashboard] - Cisco

Device Manager error in Cisco

Recently, when we try to open Device Manager to make changes on the MDS 9710 switch, the error below pops up.  It only happens to one of the sites.  The other site works fine.  We try to open device manager thru NDFC and get the same error.  

We did see a different error in the past and needed to switchover the controller in the past.  The error was "Busy network, no route, or snmpd is unresponsive."

See EMC kb 000218149.  However, it still fails to open.  

We suspect something with network but still open a ticket with Cisco.  Support runs some trace and does not see issue on the switch side and there is connectivity between switch and the client running Device Manager.  Eventually, we workaround the issue by confirming TCP is set to true for DeviceManager.bat.

set JVMARGS=%JVMARGS% -Dsnmp.preferTCP=true

We still have trouble after that.  So, we login to the switch in the other site that we don't have issue with.  Then click Device > Preference.  

Select TCP for Use SNMP on next launch. Click Apply then Ok.  

Try to use Device Manager on the switch that we have issue.  Now it works fine.  Only problem with the workaround is to have a switch that can connect to Device Manager without issue.  I have not asked support if there is any other way to force it to use TCP instead of UDP.  

NDFC bugs since deployed

NDFC was deployed in last Oct.  Couple of bugs were discovered.  

1) About 4 - 6 weeks, we will see an alert 

Elasticsearch error - 'could not fetch component status'

The bug ID is CSCwm51621.

If you follow the bug ID above, there is a solution from the forum.  For me, when that error comes up, I just reboot the appliance with "acs reboot".  The error will go away, and it won't come up until another 4 - 6 weeks.  A normal reboot is sufficient and DO NOT add any other option after "acs reboot".  

2) /logs/k8/pods 90% usage alert.  About 4 months after deployed, an alert /log/k8/pods 90% usage showed up in the Admin Console of the Nexue Dashboard.  

Support was contacted to clear the old logs.  Currently, there is no fix, and webex is required for support to clear the old logs manually.  In the future release, the log retention for the folder will be changed based on the chat with support.  

NDFC upgrade / deploy part 1 - License planning

With DCNM reaching EOL in Apr 2026, the only option left is to upgrade to NDFC.  Some site may decide to run with DCNM until the whole infrastructure is migrated to cloud.  Because we are still refreshing old MDS 9700 switches, migrating to NDFC is a must.  The main issue with firmware upgrade for existing switch is smart license.  We manage no more than 10 MDS FC switches at a time.  So, setup is kind of simple.  Still, it is more complicated compared to DCNM.

It looks like 9.2(1a) is the last version of firmware in MDS 9000 supporting legacy license / license file.  If you upgrade existing switch running older version of firmware to 9.2.2, the switch license will be changed to smart licensing auto.  Check with support.  The answer I got is those existing switch upgraded to newer firmware will be in Donor mode in the worst case.  There should be no impact to functionality according to support.  Because some of the MDS switches licenses were purchased through 3rd party vendor, and we change to another 3rd party vendor for support later, I don't know what will happen if there is licensing issue after firmware upgrade on existing switch.  So, we decide to keep existing older MDS 9700 switches with firmware 8.4(2e) until hardware refresh completes in the next 2-3 yrs.  There are a few bugs impacting firmware 8.4.  Confirm you have all the workarounds before deciding to stay in 8.4 firmware.  If you decide to upgrade existing MDS 9700 to use smart license, check with support first.   

For all the new MDS 9700, they are all shipped with firmware 9.4.x.  So, smart license is enabled auto.  Check with support and there is no license file any more on these new MDS 9700 switches.  License is installed at the factory.  

Because it is a close environment, we setup our NDFC smart licensing to offline mixed mode.  Mixed mode will support existing license file in older MDS 9700 with older 8.4 firmware.  We contact support to transfer existing legacy DCNM server license to smart license before upgrade.  At most 365 days / switch addition or decommission, I will export the license info back to Cisco software support website and then import it back to the NDFC server licensing section.  

Below are some of the Cisco Smart licensing links / doc for your information.  

Cisco Nexus Dashboard Fabric Controller Licensing FAQ - Cisco

Brownfield_Conversion_QRG

Cisco MDS 9000 Series Licensing Guide, Release 9.x - Smart Licensing Using Policy [Cisco MDS 9000 NX-OS and SAN-OS Software] - Cisco

Cisco MDS Smart Licensing Using Policy Data Sheet

DCNM login error "RequestSendFailed: EJBCLIENT000409" for the SAN Client

We have plan to build a new NDFC to manage our MDS switches.  It is more complex and will take some time to deploy.  Few challenges are resource requirement, license transfer and other new requirement.  In the meantime, we continue to use existing DCNM 11.5(4).  

Last week, suddenly, there is a login error with the java SAN client "RequestSendFailed: EJBCLIENT000409".  We have restarted the services and reboot the DCNM server.  However, it does not help.  

Do some research and most likely it is related to certificate.  We don't have our own certificate.  So, most likely, the default expired.  It is confirmed by checking the certificate expiration date from the web browser.  So, open a ticket with support and ask them to renew the certificate for 2 more years.  Then, I can login without issue.  

Migration from VMAX 40k to PMAX 8000 and 8500

Just completed the storage migration from VMAX to PMAX 8000 and 8500 before Christmas.  NDM is not an option because NDM no longer supports Solaris.  Besides, there are a number of busy databases running in both test and production.  Before cutover to PMAX occurs, the SRDF directors between VMAX and PMAX will be the bottleneck.  Also, we do not have any extra director left to config for SRDF traffic.  

Our plan is to use Storage vMotion for all VMDK.  The RDMs are for SQL DB running in ESX.  For the RDM, we use SRDF/S to start replication to PMAX in the background.  Because there is no SRDF supported from VMAX to PMAX 8500, all luns migration using SRDF will be replicated to PMAX 8000.  Then, app owner will pick a downtime to shutdown the app and the servers.  We will stop the sync after confirming no outstanding tracks.  Remove the VMAX luns from the initiator group then add the PMAX luns.    

For Solaris, if it is Oracle DB, same size or larger luns will be added to Oracle.  Then DBA will complete the balancing and drop the old VMAX luns.  For the boot luns and luns from other app, some of our Unix admin will use SRDF/S to migrate them to PMAX 8000.  Some decide to do host base migration.  For host base migration, we just provide a lun of same size or larger to the Unix admin from PMAX 8500.  

Below summarizes the general steps for the storage migration from VMAX to PMAX 8000 using SRDF/S.  Pls check your environment and test to see if additional steps are required.  If Volume Manager is used, migration should be complete with Volume Manager rather than SRDF.  

1) Change the source luns attribute in VMAX to dyn_rdf.  
2) setup SRDF/S pair from VMAX to PowerMax 8000 (put the target lun in temp target_SG)
3) during downtime, shutdown the apps and servers  
4) Confirm no outstanding tracks.  
5)     Perform SRDF split.  
6) Remove source lun from VMAX storage group   
7) Add the target luns to SG in PowerMax and remove from temp target_SG.
8) Host team completes lun mapping 
9) Delete SRDF pair with force option 
10)  Unset GCM bit if required  (symdev -sid xxx -devs xxx unset -gcm)
11)  Host team can perform rescan if setp 10 is required.  (They should see about 1MB more space for luns in step 10)  

12)  Power up the servers to validate

That way, they can always go back to the VMAX luns if backout is required.  

Note: in the past, if the source / target luns are not mapped to FE ports, sometimes will see some strange results on some of the SRDF operation.  So, I create a temp target_SG with no HBA in the IG for the target_SG's masking view.  

Isilon NDMP backup failed after NetWorker upgraded to 19.11

We need to update firmware of existing Isilon to 9.7.1.x to add new Isilon nodes as recommended by support.  Before we do that, we need to confirm about NDMP backup with Isilon.  Because we have NDMP backup of Isilon by NetWorker, NetWorker upgrade to 19.11 from 19.10.0.4 is required.  After NetWorker upgrade in Feb, some of the NDMP backups failed with error message "Hostname resolution failed ".   Multiple retries will work.  However, it is very annoying.  

After working with support, there is new change to NetWorker in 19.11.  See kb NetWorker: server upgraded to 19.11, backup fails reporting "Hostname resolution failed" | Dell US

None of the workaround in the kb above will fix the issue on Isilon.  The forward DNS lookup for Isilon are actually forwarded to Smartconnect SIP from DNS server.  Only option is to add a reverse entries to the DNS server for all the Isilon nodes that handle NDMP backup.  Refer to link SmartConnect and Reverse DNS | Dell PowerScale: Network Design Considerations | Dell Technologies Info Hub.  

So, pointer records for all Isilon nodes handling NDMP backup will be created on the DNS server for the NDMP zone name.  Once that is done, all backup is fine.  Keep in mind there is no change on the forward lookup which is still handled by Smartconnect SIP.  Not sure if there is a fix now.  

Update Isilon to 9.7.1.4 from 9.4.0.14 to add new A300, H700 and F710 nodes

Main reason for the upgrade in Jan is to replace existing A200, H500 and F800 with new Isilon nodes A300, H700 and F710 nodes.  Support recommends to update firmware of existing nodes to 9.7.1.4 before adding new Isilon nodes to the pool.  

Things went smoothly during upgrade.  So far no issue.  Right after upgrade, I need to reconfigure the following settings again.  

1) SNMP node restriction got reset.  Manually select node 1-4 again (In our environment, we only allow SNMP trap from A200 nodes.  The A200 nodes handle only NDMP backup; the only VLAN for those A200 nodes)

2) SNMP v3 is new feature and selected auto in alert channel for SNMP.  Manually select SNMP v2 for our environment

3) SMTP setting is reset back to the manual settings (nothing required since it is populated with the same SMTP info)

New features

1) IPv6 feature is new and enabled by default

2) New feature for Transfer Limit at 90% for spillover pool.  I guess it will not fill it up if pool is 90% full and there is other pool with space less than 90%.

3) Support Assist is required for SCG in future OneFS.  

4)     Firewall 

Create partition with Linux

Sometimes, for some odd reason, there is issue deleting and reforming partition on USB disk.  I have to use Linux to complete the task.  Again, this will erase everything on the USB disk.  Assume the device handle for the USB jump drive is /dev/sdb.  

1) run "sudo fdisk /dev/sdb"

2) type "o" to create new dos partition table then hit enter

3) type "n" to create a new partition then hit enter

4) hit enter if you are ok with default settings

5) type "t" then "7" to create a exFAT partition

6) type "w" to save the change then "q" to exit.  

7) run "sudo mkfs.exfat -n "label" /dev/sdb1

Isilon upgrade from 9.1.0.x to 9.4.0.14 to fix NFSv4 lock issue

We discover the NFS lock issue in OneFS 8.x with MQ.  We have MQ with message queue stores in the Isilon NFS share.  When Isilon node reboots due to maintenance, the NFS lock does not fail over to the remaining Isilon node correctly.  That causes MQ hung.  Sometimes, server requires reboot to fix the issue.  There are multiple Isilon fixes to it.  Eventually, this is fixed in 9.4.0.3.  We update to 9.4.0.14 and confirm issue is fixed.  If MQ is running in Solaris OS, pls confirm Solaris is running at least 11.4 SRU69. 

After upgrade to 9.4.0.14, some settings are reset.

1) Spillover pool reset (set to Anywhere)

2) SNMP node limit reset (we only allow SNMP alerts sent from NDMP nodes.  That is A200 in our environment).  I need to reconfigure it.  

New problem comes up after upgrade.  NDMP random backup fails with memory leak

New feature data inline introduced in 9.3 and enabled by default.  After a month of upgrade to 9.4, there is random NDMP backup failure due to memory leak.  Eventually, turn off data inline with support assistance.  Then reboot all the nodes running NDMP backup one by one to workaround the issue to clear the dedupe cache.  Permanent fix will be available in 9.7.1.x.  

Setup VLAN with TP-LINK WR1043ND v1.6 (LibreCMC)

As mentioned before, I have flashed LibreCMC on TP-LINK WR1043ND v1.6 with WiFi disabled. It is my backup router.  Because my ISP support 2 IPs, and the new cable modem does come with 2 ethernet ports, I have one port connecting to my DIR-882 (using Padavan firmware) serving the main floor and my bedroom in 2nd floor with WiFi enabled.  The 2nd port is now connected to TP-Link WR1043 in the study room.  Now, I want to create a separate VLAN for miner, so, it is separated from my test equipment.  So, I update LibreCMC to 1.5.14.  It requires the "Keep Settings" to be unchecked for the upgrade.  Please check the link below before proceed.  Since it was a backup router, it does not matter for me if the settings are wiped.  

Releases - Gogs (librecmc.org)

There is not many guides in the internet for LibreCMC.  Since it is based on OpenWrt 19.x, I check the youtube guide below.  

How to Create a VLAN - A Beginner's Guide // OpenWrt Router (Up to 19.x) - YouTube

How to configure OpenWrt as Firewall for your home network and Guest Wifi and IPTables explained - YouTube

First step is to go to Network > Switches.  You will see VLAN 1 and 2 populated already.  Now, add a new switches with VLAN 5.  I pick LAN port 3 for VLAN 5 which will be used by the miner.  VLAN 1 will be used for test equipment (LAN port 1, 2 4). 

Second step is to go to Network > Interfaces.  Click "Add new Interface" to add a new interface named  VLAN_5.  Below are the rest of settings.

Protocol: static addresses

For my setup, "Create a bridge over Multiple Interfaces" is unchecked since WiFi is disable.  

Cover the following interfaces: select eth0.5 (automatically created when VLAN 5 is created in previous step)

Then, go to Network > Interfaces again.  Click on the VLAN_5 interface and apply the settings below.

Protocol: Static address

Bring up on Boot: checked

IPv4 address: default gateway for subnet VLAN 5

IPv4 subnet mask: select the subnet mask

Then scroll down to DHCP server section.

Select the range for DHCP client IP address and the lease time. 

Click on Advance Settings under DHCP section.  Make sure Dynamic DHCP is checked.  

Now, click on Physical Settings tab.  In my case, Bridge Interface is unchecked since WiFi is turned off and the correct Interface eth0.5 is selected for VLAN 5.  

Next, go to Firewall Settings Tab.  Create a new firewall zone called VLAN5 for VLAN 5.  

Lastly is to config firewall.  For VLAN_5, I don't want the miner to reach the router other than getting IP and DNS resolution but will have access to internet.  Also, it cannot reach to VLAN 1.  

So, setup VLAN5 zone to allow VLAN5 forwarding to WAN to get internet access.  Input and forward are set to reject and Output is set to accept.  

To allow client (miner) in VLAN_5 to reach to router for DHCP and DNS, setup a rule to "Accept Input" from VLAN_5 to router IP at 53, 67 and 68.

There are other exception rules predefined.  You can disable them based on your requirement.  

Setup Tumbleweed as physical host for VirtualBox

I decide to try Tumbleweed on the same old FX-8320 host.  Installation is similar to LEAP 15.4.  For GUI option, I choose Generic Desktop instead of XFCE Desktop.  XFCE is running quite smoothly in LEAP 15.4.  I decide to try Generic Desktop this time since it is only a physical host.  

Follow the same link below, I try to install driver version G05 for my old GT630 (does not work with LEAP 15.4).  

SDB:NVIDIA drivers - openSUSE Wiki 

Reboot the host and it suggests to install the previous version G04.  Follow the wiki link above and install G04.   

 

Below are my software pick for the host.  

File Manager: PCmanFM

Terminal: termit

Text Editor: Leafpad

Graphics: Pinta (take and edit basic screenshot)

GRsync: sync my USB drive for backup.  

USB wireless 8812bu adapter is discovered correctly.  No driver installation is required.  

Only issue is NetworkManager icon is not available for Generic Desktop.  First I install wicked and other tools.  It works fine but I decide to try to get NetworkManager working.  Then I remove wicked and not sure why, there is no networking at all.  So, I rollback the system to the point before wicked is installed.  

Once NetworkManager-Connection-Editor is installed, I have the simple GUI to manage my network connection.  If your connection does not change much, wicked is just fine. 

Then install VirtualBox.  Reference link is How to install Virtualbox on OpenSUSE Tumbleweed & Leap - Linux Shout (how2shout.com.

sudo zypper install virtualbox

I find out 7.0.10 is installed.  

Once completed, add the user account to group vboxusers

sudo gpasswd -a userID vboxusers

Then, logoff and log back in.

Open VirtualBox.  If you are asked to enable USB passthrough, just click Enable.  

Download the VirtualBox extension pack from VirtualBox site.  Same as before, it does not work with GUI installation.  I install it thru CLI (see command below)

VBoxManage extpack install extpack_7.0.10

I decide to go with KVM/QEMU this time.  Will create a separate post for that. 

Isilon multiscan moves data between tiers unexpectedly

 After Isilon was implemented 2.5 yrs ago, we decided to stop SmartPool jobs and use combination of IndexUpdate and FilePolicy job to save resources on the Isilon.  We have 4 SSD nodes handling front end traffic, 4 hybrid nodes handling replication plus some test traffic and 4 SATA nodes handling the backup.  Initially, SATA nodes were deployed with insufficient memory and caused couple of outages during NDMP backup.  After max the memory on SATA nodes, there are no more issue after that.  

Because we want to keep as much data as possible on SSD, we won't run FilePolicy until SSD used space is over 70%.  After firmware update (nodes reboot), we saw data move between Tiers with multiscan kicking in auto.  That's about 2 yrs ago.  Checked with support and they said it's ok to kill it.  So, we just killed the Mutliscan job.  Recently, we finally had our first disk failed on the production Isilon and Multiscan was kicked in auto.  Same as before, we saw data moving unexpectedly to SATA tier.  I suspected if that's related to FilePolicy.  However, I was told it should not by support.  After case with support for a long time, we finally got the suggestion from higher level of support to run SmartPool job periodically.  So, I play with the DR Isilon since it is not busy.  Using IndexUpdate and FilePolicy job to move the data and compare the tier usage multiple times, each tier finally reached to the utilization I want 60-70% for SSD and below 60% for SAS pool (the spillover pool).  Then, kick off MultiScan job.  Now, I don't see any more data move between tiers with MultiScan job.  

In the future, I will adjust the FilePolicy each month and then kick off the SmartPool job once a month just in case Multiscan is started due to failed HDD or node replacement.

===========================================================================

Update Sep 25, 2023

For Production Isilon, after IndexUpdate completes, I run FilePolicy then Multiscan job.  I still see data moves to SATA pool.  So, I run IndexUpdate -> FilePolicy -> SmartPool job.  Same things.  Looks like there is discrepancy between IndexUpdate + FilePolicy and SmartPool job.  

This time, I just run SmartPool job and kill it once SATA pool reaches 85%.  Then adjust the FilePolicy and rerun it SmartPool job.  After 3 tries, I finally see the results I want.  Why there is a discrepancy between FilePolicy and SmartPool job, I have no idea.  

From now on, I will adjust the FilePolicy once every 2 months and run the SmartPool job.  For DR Isilon, since it has far less data, I will adjust FilePolicy once every 3-4 month and then run SmartPool job. 

Cable Modem TC4400 overheated?

Switch to TC4400 cable modem for a few months because DOCSIS 3.0 will not be supported by my cable company.  This is the only one not using Puma chipset and supported by my cable company.  It worked fine until recently.  The internet connection drops few times day.  When I touch it, the modem is really hot.  The quick fix is to put a little 15mm by 15mm fan on the top of modem to draw the heat away from it.  Now, it is much better and has not experienced any more connection drop.  

How long to quick format a lun in Windows

 I always get that question for the SQL or file cluster RDM disks.  Windows admin and DBA thought the format hung but it indeed took long time to quick format a lun.  Last week, Windows admin complaint it took approx an hour to quick format a 7TB lun in EMC PMAX for a Windows 2016 / 2019 VM.  

I just completed a test this morning.  For a physical Windows 2016 server, it took about 10-15 min approx to quick format a Windows 1 TB lun in PMAX.  Do some research in the internet.  The result provided by partitionwizard.com suggests it indeed takes long time to format a big lun (see below also).  

• How Long Does It Take to Format a 1TB Hard Drive: Performing a Quick Format on a 1TB hard drive takes about 20 minutes. If you select the Full Format, it could take you up to 1 hour.
• How Long Does It Take to Format a 2TB Hard Drive: Again, we perform a Quick Format on a 2TB hard drive, it can be done in about 30 minutes. However, a Full Format can take up to 3 hours. If this hard drive stores a chunk of data, it could take you a half day.
• How Long Does It Take to Format a 4TB Hard Drive: To a certain degree, 4TB is a large hard disk that will take quite a long time to format. So, you’d better select a Quick Format if you want to save time. This is because fully formatting a 4TB hard drive can take you a whole day and even more

This give you an idea even quick format in Windows will still take some time.  

Our Wintel team will test the format time following EMC article 000062689 on the PMAX.

The cause is "trim and unmap" feature is on.  So, just temp turn it off before formatting new lun and enable it back once done.  

On the windows host, disable the SCSI TRIM and Unmap feature for the duration of the format. Use fsutil command from the command line

1) To verify the current setting, using a Windows CMD window on the Host, run:   

fsutil behavior query DisableDeleteNotify

DisableDeleteNotify=0 -indicates the 'Trim and Unmap' feature is on (enabled)

DisableDeleteNotify=1 -indicates the 'Trim and Unmap' feature is off (disabled)

2) To disable, issue the command:   

fsutil behavior set DisableDeleteNotify 1

3) Once formatting is complete, re-enable the feature using command:   

fsutil behavior set DisableDeleteNotify 0

It may impact Linux as well.  See thread mkfs is extremely slow. 

To run mkfs without trim, use the -K option on XFS and -E nodiscard on ext4

XFS

mkfs.xfs -K /dev/sdx 

EXT4

mkfs.ext4 -E nodiscard 

Warning: Only use -K or -E on new volumes with no existing data.

Using the -K or -E options on drives with existing data, will cause the space to be wasted until the data is overwritten.

  

Install Open VM Tools on Linux VM

Depending on the distro of Linux, I have tough time to install VM Tools on some of them.  The only other option is to install open vmtools.  

Follow VMware instructions to install the open vmtools.

If GUI is required on the VM and assume it is Debian distro, install open-vm-tools-desktop.  

sudo apt-get install open-vm-tools-desktop

Otherwise open-vm-tools will be sufficient.

sudo apt-get install open-vm-tools  

No Port Down email alert from DCNM

We used to have SNMP to monitor all FC port down event.  However, it causes a lot of un-necessary calls from NOC after hours.  Because we have most of the prod servers setup with 4 HBAs, we decide to use SMTP alert to email to storage team members only.  However, it never seems to work with DCNM.  We did see Port up alert but not for Port down regardless of what settings we choose.  

After opening a ticket with support, we were told the setting was disable by default.  If you want to receive email alerts for port down event, go to Server Properties of DCNM.  Look for event.linkDown.log and set it to True.  By default, it is set to False.  Then restart the DCNM services and port down alerts will be sent to your SMTP server depending on your setup.  

Cisco bug CSCvz61883

Check the bug info at Cisco site for CSCvz61883 / EMC article 000197332 and it applies to the MDS 9700 32G FC modules in our environment.  The DS-X9648-1536K9 linecard in MDS 9700 can be affected at almost exactly 468 days of uptime.  

Our MDS 9710 with 32G linecard are running 8.1(1a) with supervisor 3 and 8.4(1a) with supervisor 4.  Runs the command below against the 32G linecard in MDS9710 to see module uptime.  In our environment, they reach 426 and 417 days respectively.    

slot x show system uptime   

8.4(2d) does provide the fix to the issue but it is quite new.  Open a ticket with support and the recommendation is to upgrade to 8.4(2c).  This will reset the module uptime back to zero and buy another 400+ days.  With 8.4(2), there is a workaround to reset the uptime for the device non-disruptively.  You can consider another firmware update after 400 days on the FC switch.  

Finished the upgrade to 8.4(2c) last week.  Upgrade to 8.4(2c) on MDS 9710 with supervisor 3 is much faster than MDS 9710 with supervisor 4.  

=================================================================

Updated in late May 2023

Another reason we never considered 8.4(2d) was due to CSCwb29379 (see release notes 8.4(2e) below) 

Cisco MDS 9000 Series Release Notes, Release 8.4(2e)

8.4(2e) was available for quite some time now.  So, we upgraded to 8.4(2e) couple weeks ago to fix this problem permanently.