Unisphere 9.x with VMAX 40k

Recently, I update Unisphere and Solutions Enabler to 9.0 for the VMAX 40k.  Unisphere 8.4 will not be supported after May 2020.  Few things are not supported for VMAX 40k in Unisphere 9.0.  Belows are the things I don't like about new Unisphere 9.

1. I cannot export list of luns in storage group.  I guess EMC wants customer to use ViPR SRM to run report (not sure if this is only applied to VMAX 40k)
2. No more provision template for VMAX 40k and that is not convenient.  
3. Cannot change FAST priority in the GUI any more for SG of VMAX40k
4. Heatmap is not shown in a single page.  

Heat Map in 8.4

Heat Map in 9.0

      5.   In Unisphere 8.4, if you want to delete meta members after dissolve a meta lun, you can select the option to delete meta members.  However, it will only delete meta head in Unisphere 9.  It will be fixed in Unisphere 9.1 according to support.  Because there is no more meta from VMAX, I suspect the developer forget about the VMAX 40k.    

6. There is a limitation with Unisphere 9.0.  Only the first 1024 luns will be shown.  You can use the filter to display the luns after the first 1024.  The filter for lun label does not work too well in 8.x.  However, once it is fixed in version 9, now, only the first 1024 luns will be shown.  See EMC kb Unisphere for PowerMax: Maximum viewable amount is 1024 volumes(000529643)

There was a bug in earlier version of Unisphere 8.x.  When you try to delete a meta lun under Virtual lun, it will crash the Unisphere GUI for VMAX 40k.  It is fixed in Unisphere 8.4.  Glad it is passed to Unisphere 9.0.  Also, if you try to create metalun from Unisphere, striped is populated auto and cannot be changed.  I have not tried to create concatenated lun with CLI in SE 9.0.

Just a reminder don't forget to change the smc password after installing Unisphere.

For VMAX 40k, you cannot change the Unisphere smc password running in SP.  So, make sure to restrict IP access to SP from the network. 

========================================================================
Just upgrade to Unisphere 9.1 (12/2019) 
It does fix the issue mentioned before for dissolve and delete meta luns.  However, it generates 3 new problems.

a) AD login does not work after upgrade.  (using local account at the moment)
b) Cannot customize performance alert for each thin pool / disk group.  (no workaround)
c) Dash Board reduces the overall health by 20 points because SSD pool is 95% full (recommendation  by vendor is to reserve only 1% of space given no lun is bounded / pinned to SSD and only thin provision is used).  This is not an issue.  Just ignore it. 

Hopefully, next patch fixes it. 

Also database in 9.1 is different.  Upgrade fails at database upgrade.  Since I ran all the monthly reports, I actually need to uninstall 9.1 and then install again in another folder.  So, it is a fresh installation for me.  
========================================================================

Update to Unisphere 9.1.0.20 (11/2020)  

AD login issue is still not resolved.  Still using local account.  

Alert customization for each Thin Pool / Disk Group seems fixed if I set it up in IE (won't work with Chrome).   

I doubt the last one mentioned before will be fixed.  Dash Board still reduce overall health by 20 points for SSD usage over 95%.  I will probably stay in this version until migration to PowerMax.  

Performance between different OS for Raspberry pi 4

Raspup 8.2 is the first one I installed in Raspberry pi 4 (4GB RAM).  It is ok but I am not happy with the performance for web browsing.  So, I tried Gentoo and Raspberry Pi OS.  I got the Raspberry pi 4 as a desktop replacement for the old laptop for my parents.  They only use browser or goto youtube.  

Definitely, Raspberry Pi OS is slightly faster than Gentoo.  Raspup 8.2 performance is the worst among the three.  Since they use it for browsing, I have not spent time to do research on how to make it faster.  So, I only use default settings.  

You can find the OS links below.  

Raspup | Home (eezy.xyz)

Releases · sakaki-/gentoo-on-rpi-64bit · GitHub

Raspberry Pi OS – Raspberry Pi

AppSync login page won't load with 404 error

 After Windows patching and AppSync server reboot, the management page of AppSync won't load.  It only show the 404 error Not found.  Restart server makes no difference.  Check from support page and find the kb 501522  https://support.emc.com/kb/501522

I do see the three files with undeployed below.  So follow the kb and the issue is fixed.  

apollo.ear.undeployed

archway-ear.ear.undeployed

remotex.rar.undeployed

Below is kb501522 from EMC support site.  

Cause

This may be seen if the Appsync Server services are Started and Stopped in quick succession. There are a number of files, such as E:\EMC\AppSync\jboss\.\standalone\..\applications\apollo.ear which are deployed by the Appsync Server during startup. If the service is stopped before these files are fully deployed the Appsync Server may begin to throw this error.

When we check the location C:\EMC\AppSync\jboss\applications we should see the following 6 files in a healthy system: 

• Apollo.ear
• apollo.ear.deployed
• archway-ear.ear
• archway-ear.ear.deployed
• remotex.rar
• remotex.rar.deployed

When we check on a system showing the "Error 404 not found" error we should find some of these files marked as undeployed, eg "apollo.ear.undeployed ".

Workaround

In order to resolve this issue we have to get the Appsync Server service to deploy the .EAR files. Follow the procedure outlined below to accomplish this. 
When we find undeployed files in the C:\EMC\AppSync\jboss\applications folder perform below mentioned steps to resolve:

1. Stop all the services.
2. Take backup of the C:\EMC\AppSync\jboss\applications.
3. Rename Apollo.ear.undeployed to Apollo.ear.dodeploy.
4. Perform the same steps as step 3 in case any other EAR files are also shown as undeployed.
5. Start the services.
6. Check the status for .EAR files again at the location C:\EMC\AppSync\jboss\applications.

i. Apollo.ear

ii. apollo.ear.deployed

iii. archway-ear.ear

iv.archway-ear.ear.deployed

v.remotex.rar

vi.remotex.rar.deployed

7. Try to log in into appsync GUI after 5 minutes and you should be able to log in.

Update Cisco DCNM to 11.3(1) due to security Vulnerabilities

Because of Cisco DCNM security issues (see below), just update DCNM to 11.3(1).  Update should be done ASAP.

I am not using the appliance and it is running on Windows with version 11.1(1).  It is running with Oracle Express 11g and managing only MDS FC switches.  We don't use advance features like SAN Insight and it is a standalone Windows server.  Upgrade step is pretty straight forward.

https://www.cbronline.com/data-centre/cisco-data-center-network-manager/

Setting up PuppyLinux on my Raspberry

Just receive my Raspberry pi 4 with 4GB of memory.  Decide to install Raspup 8.2 on it.  Details instructions can be found in Raspup page.

I copy the Windows instructions below.

1. Download Raspup image from the Downloads page and check the checksum.
2. Extract the image file from the downloaded .zip file, so you now have "raspup-XXXXXX.img".
3. Insert the SD card into your SD card reader and check what drive letter it was assigned. You can easily see the drive letter (for example G:) by looking in the left column of Windows Explorer. You can use the SD Card slot (if you have one) or a cheap Adapter in a USB slot.
4. Download the Win32DiskImager utility (it is also a zip file). You can run this from a USB drive.
5. Extract the executable from the zip file and run the Win32DiskImager utility; you may need to run the utility as Administrator! Right-click on the file, and select 'Run as Administrator'
6. Select the image file you extracted above.
7. Select the drive letter of the SD card in the device box. Be careful to select the correct drive; if you get the wrong one you can destroy your data on the computer's hard disk! If you are using an SD Card slot in your computer (if you have one) and can't see the drive in the Win32DiskImager window, try using a cheap Adapter in a USB slot.
8. Click Write and wait for the write to complete.
9. Exit the imager and eject the SD card.
10. You are now ready to plug the card into your Raspberry Pi.
11. In Windows, the SD card will appear only to have a fairly small size once written - about 512 MB. This is because most of the card has a partition that is formatted for the Linux operating system that the Raspberry Pi uses which is not visible in Windows. If you don't see this small directory with files such as kernel.img then the copy may not have worked correctly.

PuppyLinux is my favourite Linux and it is a nice desktop for my dad to browse the net.

To read Chinese, just install package fonts_arphic_uming.  

To create Shortcut, go to /usr/share/applications and drag the icon to the Desktop.

Equallogic dirty cache

I used to manage a large number of Equallogic 6510.  They have bought them for production used because of insufficient budget for enterprise storage.  It is just an entry level array with very limited redundancy.  Controllers are active / standby and it takes 21 s to failover when it is completely idle.  So, you can imagine how long it will take if there is heavy IOPs.

It is a pain to update firmware since it takes some time for controller to failover.  Linux and Unix will not like that.  If you used it for production, it will be really hard to get downtime.

One time, there was a bug and both controller panic.  After it starts back up, the management interface is not reachable.  Check serial connection and see the following.  Even though the controllers panic, it still displayed a msg "This is a POWER FAILURE RECOVERY".  Also, it showed RAID LUN not recoverable.  Keep looking down, sounds like we actually have a dirty cache stuck in the memory.  This normally happens with power failure.  

When I try to login from serial port, it shows the array is not even configured.  Obviously answer No when asked to config the array.  

Contact support from that point and it is indeed a dirty cache problem.  Talk to support and confirm if the dirty cache stuck, user will lose management interface access.  Also, only later model of EQL support port failover to standby controller.  If both interfaces from the active controllers die, the interface will not failover to standby controller interface.  You will need to do a manual failover.  That's why I don't suggest them for production. 

Clear the cache and reboot the array.  Everything is normal from that point.  Only thing you lost is the data in the stuck cache.  Luckily, there is no database running in those arrays.  The uncorrectable sectors are empty space. 

Earlier version of firmware especially version 5 and 6 are problematic.  Lots of problem.  After the latest patch of version 7 installed, we see stability from that point.  However, new enterprise arrays were installed, and these units were used for backup / archiving.  Now, they were all retired.  

Install IE and flash in Windows server 2016

Find the link below on how to reinstall IE in Windows 2012 server.

https://www.codeproject.com/Articles/1158545/Re-install-Internet-Explorer-in-Windows-Server-R

To install only IE in Windows Server 2016, just run the command below and reboot.  

dism /online /enable-feature:"Internet-Explorer-Optional-amd64"

If you need to enable flash for IE, you can run the command below.  

dism /online /add-package /packagepath:"C:\Windows\servicing\Packages\Adobe-Flash-For-Windows-Package~31bf3856ad364e35~amd64~~10.0.14393.0.mum"

iDRAC version 8 virtual console only works with IE

I have no trouble accessing iDRAC v8 GUI using IE 11, Chrome and Firefox.  It is a R630 server.  However, I found out I can only access Virtual Console using IE 11.  Chrome v76 and Firefox v67 won't work for Virtual Console.

I guess try to use other browser if you cannot access Virtual Console.

AppSync with VMAX 40k

AppSync 3.5 was installed couple years ago to protect our SQL application.  Basically, AppSync agent was installed in SQL cluster.  The SQL clusters were running as VMs and the databases were residing in RDM.

Kept getting VSS error that it takes more than 10s to create VSS.  That was what MS supported for VSS.  If it took more than 10s, the VSS creation step would fail.

Installed latest Solution Enabler 8.4.x.x available at the time and no change.  Since the array was VMAX 40k, DNS was not a factor.  Eventually, version Version 3.5.0.1_URM00111091_PRELIM_R2  fix the bug VSS 10s delay bug with VMAX 40k.

Couple things I don't like about the AppSync.
1)  Mounting and Dismounting SQL VMFS takes long time and does not work well.
2)  Need to reserve an extra copy of luns in the pool   
3)  If info is not sync, AppSync does not know what to do.  So, manually dismount copy from mount host will cause problem because AppSync does not know the luns are dismount.  Eventually, contacting support to clean up is required.

------------------------------------------------------------------------------------------------------------

Now, 3 yrs later, AppSync 3.9 was setup for proof of concept few months ago.  This time, we have SQL running in VMFS.  We encounter same issue as before.  Mounting copy to mount host is very slow and timeout.  Eventually, we keep our design for SQL as before in RDM.  Things are working smoothly with RDM as expected.   

We are still using the AppSync 3.5 now until we migrate to AppSync 3.9 next yr.  Recently, SQL team change the DB structures and instead of a few larger database, we have a lot of smaller database to be protected.  The AppSync host plugin service has memory leak problem.  So, we have to setup a process to bounce the service once every 3 days.  Hopefully, this won't happen in version 3.9.

DCNM version 11.1(1) and bug ID CSCvf99665

Recently build a new DCNM 11.1(1) box in Windows 2016 to replace the existing 7.2(3) because the old one is running Windows 2008 R2.  Major difference is HTML5 and the webclient is a lot faster and most of the work even port channel can be completed in the GUI (I have not tried that yet).  If you don't like to use the new webclient to complete your zoning, you can still use the old FM. 

Besides, I use the Oracle Express for the db of DCNM since very 7.2.  The performance is better than the POSTGRESQL.  

You can follow the Oracle link below to have some basic knowledge of Oracle Express.

https://docs.oracle.com/cd/E17781_01/server.112/e18804/toc.htm

If you use SolarWind as your TFTP server, make sure .NetFramework 3.5 is required.  See link below on how to enable it in Windows 2016 server.

http://www.virtubytes.com/2017/02/03/install-net-framework-3-5-server-2016/

After new DCNM server is in production, I plan for the firmware update on all the FC switches in the fabric.  However, I find out both of the MDS9710 are affected by Cisco bug ID CSCvf99665

It show an invalid IPV6 IP address in the mgmt/0 interface and has a zero length subnet mask.

Example:

::148.237.143.255/0

Suggestion from support

(1)Open a case and get Cisco TAC to send you a DPLUG file that will be downloaded and run on the switch.  We would run the DPLUG, do a 'copy r s' to save the configuration, then do a 'system switchover' and run 'copy r s' again.

(2)Upgrade to 8.1(1a).  After the upgrade has completed, do a 'copy r s',  do a 'system switchover' and after both supervisors are back up, then do another 'copy r s'.

On the safe side, we choose the 1st option and have support apply the DPLUG file.  Everything is fine.  Then, we update the firmware to 8.1(1a).