Setup VLAN with TP-LINK WR1043ND v1.6 (LibreCMC)

As mentioned before, I have flashed LibreCMC on TP-LINK WR1043ND v1.6 with WiFi disabled. It is my backup router.  Because my ISP support 2 IPs, and the new cable modem does come with 2 ethernet ports, I have one port connecting to my DIR-882 (using Padavan firmware) serving the main floor and my bedroom in 2nd floor with WiFi enabled.  The 2nd port is now connected to TP-Link WR1043 in the study room.  Now, I want to create a separate VLAN for miner, so, it is separated from my test equipment.  So, I update LibreCMC to 1.5.14.  It requires the "Keep Settings" to be unchecked for the upgrade.  Please check the link below before proceed.  Since it was a backup router, it does not matter for me if the settings are wiped.  

Releases - Gogs (librecmc.org)

There is not many guides in the internet for LibreCMC.  Since it is based on OpenWrt 19.x, I check the youtube guide below.  

How to Create a VLAN - A Beginner's Guide // OpenWrt Router (Up to 19.x) - YouTube

How to configure OpenWrt as Firewall for your home network and Guest Wifi and IPTables explained - YouTube

First step is to go to Network > Switches.  You will see VLAN 1 and 2 populated already.  Now, add a new switches with VLAN 5.  I pick LAN port 3 for VLAN 5 which will be used by the miner.  VLAN 1 will be used for test equipment (LAN port 1, 2 4). 

Second step is to go to Network > Interfaces.  Click "Add new Interface" to add a new interface named  VLAN_5.  Below are the rest of settings.

Protocol: static addresses

For my setup, "Create a bridge over Multiple Interfaces" is unchecked since WiFi is disable.  

Cover the following interfaces: select eth0.5 (automatically created when VLAN 5 is created in previous step)

Then, go to Network > Interfaces again.  Click on the VLAN_5 interface and apply the settings below.

Protocol: Static address

Bring up on Boot: checked

IPv4 address: default gateway for subnet VLAN 5

IPv4 subnet mask: select the subnet mask

Then scroll down to DHCP server section.

Select the range for DHCP client IP address and the lease time. 

Click on Advance Settings under DHCP section.  Make sure Dynamic DHCP is checked.  

Now, click on Physical Settings tab.  In my case, Bridge Interface is unchecked since WiFi is turned off and the correct Interface eth0.5 is selected for VLAN 5.  

Next, go to Firewall Settings Tab.  Create a new firewall zone called VLAN5 for VLAN 5.  

Lastly is to config firewall.  For VLAN_5, I don't want the miner to reach the router other than getting IP and DNS resolution but will have access to internet.  Also, it cannot reach to VLAN 1.  

So, setup VLAN5 zone to allow VLAN5 forwarding to WAN to get internet access.  Input and forward are set to reject and Output is set to accept.  

To allow client (miner) in VLAN_5 to reach to router for DHCP and DNS, setup a rule to "Accept Input" from VLAN_5 to router IP at 53, 67 and 68.

There are other exception rules predefined.  You can disable them based on your requirement.  

Setup Tumbleweed as physical host for VirtualBox

I decide to try Tumbleweed on the same old FX-8320 host.  Installation is similar to LEAP 15.4.  For GUI option, I choose Generic Desktop instead of XFCE Desktop.  XFCE is running quite smoothly in LEAP 15.4.  I decide to try Generic Desktop this time since it is only a physical host.  

Follow the same link below, I try to install driver version G05 for my old GT630 (does not work with LEAP 15.4).  

SDB:NVIDIA drivers - openSUSE Wiki 

Reboot the host and it suggests to install the previous version G04.  Follow the wiki link above and install G04.   

 

Below are my software pick for the host.  

File Manager: PCmanFM

Terminal: termit

Text Editor: Leafpad

Graphics: Pinta (take and edit basic screenshot)

GRsync: sync my USB drive for backup.  

USB wireless 8812bu adapter is discovered correctly.  No driver installation is required.  

Only issue is NetworkManager icon is not available for Generic Desktop.  First I install wicked and other tools.  It works fine but I decide to try to get NetworkManager working.  Then I remove wicked and not sure why, there is no networking at all.  So, I rollback the system to the point before wicked is installed.  

Once NetworkManager-Connection-Editor is installed, I have the simple GUI to manage my network connection.  If your connection does not change much, wicked is just fine. 

Then install VirtualBox.  Reference link is How to install Virtualbox on OpenSUSE Tumbleweed & Leap - Linux Shout (how2shout.com.

sudo zypper install virtualbox

I find out 7.0.10 is installed.  

Once completed, add the user account to group vboxusers

sudo gpasswd -a userID vboxusers

Then, logoff and log back in.

Open VirtualBox.  If you are asked to enable USB passthrough, just click Enable.  

Download the VirtualBox extension pack from VirtualBox site.  Same as before, it does not work with GUI installation.  I install it thru CLI (see command below)

VBoxManage extpack install extpack_7.0.10

I decide to go with KVM/QEMU this time.  Will create a separate post for that. 

Isilon multiscan moves data between tiers unexpectedly

 After Isilon was implemented 2.5 yrs ago, we decided to stop SmartPool jobs and use combination of IndexUpdate and FilePolicy job to save resources on the Isilon.  We have 4 SSD nodes handling front end traffic, 4 hybrid nodes handling replication plus some test traffic and 4 SATA nodes handling the backup.  Initially, SATA nodes were deployed with insufficient memory and caused couple of outages during NDMP backup.  After max the memory on SATA nodes, there are no more issue after that.  

Because we want to keep as much data as possible on SSD, we won't run FilePolicy until SSD used space is over 70%.  After firmware update (nodes reboot), we saw data move between Tiers with multiscan kicking in auto.  That's about 2 yrs ago.  Checked with support and they said it's ok to kill it.  So, we just killed the Mutliscan job.  Recently, we finally had our first disk failed on the production Isilon and Multiscan was kicked in auto.  Same as before, we saw data moving unexpectedly to SATA tier.  I suspected if that's related to FilePolicy.  However, I was told it should not by support.  After case with support for a long time, we finally got the suggestion from higher level of support to run SmartPool job periodically.  So, I play with the DR Isilon since it is not busy.  Using IndexUpdate and FilePolicy job to move the data and compare the tier usage multiple times, each tier finally reached to the utilization I want 60-70% for SSD and below 60% for SAS pool (the spillover pool).  Then, kick off MultiScan job.  Now, I don't see any more data move between tiers with MultiScan job.  

In the future, I will adjust the FilePolicy each month and then kick off the SmartPool job once a month just in case Multiscan is started due to failed HDD or node replacement.

===========================================================================

Update Sep 25, 2023

For Production Isilon, after IndexUpdate completes, I run FilePolicy then Multiscan job.  I still see data moves to SATA pool.  So, I run IndexUpdate -> FilePolicy -> SmartPool job.  Same things.  Looks like there is discrepancy between IndexUpdate + FilePolicy and SmartPool job.  

This time, I just run SmartPool job and kill it once SATA pool reaches 85%.  Then adjust the FilePolicy and rerun it SmartPool job.  After 3 tries, I finally see the results I want.  Why there is a discrepancy between FilePolicy and SmartPool job, I have no idea.  

From now on, I will adjust the FilePolicy once every 2 months and run the SmartPool job.  For DR Isilon, since it has far less data, I will adjust FilePolicy once every 3-4 month and then run SmartPool job. 

Cable Modem TC4400 overheated?

Switch to TC4400 cable modem for a few months because DOCSIS 3.0 will not be supported by my cable company.  This is the only one not using Puma chipset and supported by my cable company.  It worked fine until recently.  The internet connection drops few times day.  When I touch it, the modem is really hot.  The quick fix is to put a little 15mm by 15mm fan on the top of modem to draw the heat away from it.  Now, it is much better and has not experienced any more connection drop.  

How long to quick format a lun in Windows

 I always get that question for the SQL or file cluster RDM disks.  Windows admin and DBA thought the format hung but it indeed took long time to quick format a lun.  Last week, Windows admin complaint it took approx an hour to quick format a 7TB lun in EMC PMAX for a Windows 2016 / 2019 VM.  

I just completed a test this morning.  For a physical Windows 2016 server, it took about 10-15 min approx to quick format a Windows 1 TB lun in PMAX.  Do some research in the internet.  The result provided by partitionwizard.com suggests it indeed takes long time to format a big lun (see below also).  

• How Long Does It Take to Format a 1TB Hard Drive: Performing a Quick Format on a 1TB hard drive takes about 20 minutes. If you select the Full Format, it could take you up to 1 hour.
• How Long Does It Take to Format a 2TB Hard Drive: Again, we perform a Quick Format on a 2TB hard drive, it can be done in about 30 minutes. However, a Full Format can take up to 3 hours. If this hard drive stores a chunk of data, it could take you a half day.
• How Long Does It Take to Format a 4TB Hard Drive: To a certain degree, 4TB is a large hard disk that will take quite a long time to format. So, you’d better select a Quick Format if you want to save time. This is because fully formatting a 4TB hard drive can take you a whole day and even more

This give you an idea even quick format in Windows will still take some time.  

Our Wintel team will test the format time following EMC article 000062689 on the PMAX.

The cause is "trim and unmap" feature is on.  So, just temp turn it off before formatting new lun and enable it back once done.  

On the windows host, disable the SCSI TRIM and Unmap feature for the duration of the format. Use fsutil command from the command line

1) To verify the current setting, using a Windows CMD window on the Host, run:   

fsutil behavior query DisableDeleteNotify

DisableDeleteNotify=0 -indicates the 'Trim and Unmap' feature is on (enabled)

DisableDeleteNotify=1 -indicates the 'Trim and Unmap' feature is off (disabled)

2) To disable, issue the command:   

fsutil behavior set DisableDeleteNotify 1

3) Once formatting is complete, re-enable the feature using command:   

fsutil behavior set DisableDeleteNotify 0

It may impact Linux as well.  See thread mkfs is extremely slow. 

To run mkfs without trim, use the -K option on XFS and -E nodiscard on ext4

XFS

mkfs.xfs -K /dev/sdx 

EXT4

mkfs.ext4 -E nodiscard 

Warning: Only use -K or -E on new volumes with no existing data.

Using the -K or -E options on drives with existing data, will cause the space to be wasted until the data is overwritten.

  

Install Open VM Tools on Linux VM

Depending on the distro of Linux, I have tough time to install VM Tools on some of them.  The only other option is to install open vmtools.  

Follow VMware instructions to install the open vmtools.

If GUI is required on the VM and assume it is Debian distro, install open-vm-tools-desktop.  

sudo apt-get install open-vm-tools-desktop

Otherwise open-vm-tools will be sufficient.

sudo apt-get install open-vm-tools  

No Port Down email alert from DCNM

We used to have SNMP to monitor all FC port down event.  However, it causes a lot of un-necessary calls from NOC after hours.  Because we have most of the prod servers setup with 4 HBAs, we decide to use SMTP alert to email to storage team members only.  However, it never seems to work with DCNM.  We did see Port up alert but not for Port down regardless of what settings we choose.  

After opening a ticket with support, we were told the setting was disable by default.  If you want to receive email alerts for port down event, go to Server Properties of DCNM.  Look for event.linkDown.log and set it to True.  By default, it is set to False.  Then restart the DCNM services and port down alerts will be sent to your SMTP server depending on your setup.  

Cisco bug CSCvz61883

Check the bug info at Cisco site for CSCvz61883 / EMC article 000197332 and it applies to the MDS 9700 32G FC modules in our environment.  The DS-X9648-1536K9 linecard in MDS 9700 can be affected at almost exactly 468 days of uptime.  

Our MDS 9710 with 32G linecard are running 8.1(1a) with supervisor 3 and 8.4(1a) with supervisor 4.  Runs the command below against the 32G linecard in MDS9710 to see module uptime.  In our environment, they reach 426 and 417 days respectively.    

slot x show system uptime   

8.4(2d) does provide the fix to the issue but it is quite new.  Open a ticket with support and the recommendation is to upgrade to 8.4(2c).  This will reset the module uptime back to zero and buy another 400+ days.  With 8.4(2), there is a workaround to reset the uptime for the device non-disruptively.  You can consider another firmware update after 400 days on the FC switch.  

Finished the upgrade to 8.4(2c) last week.  Upgrade to 8.4(2c) on MDS 9710 with supervisor 3 is much faster than MDS 9710 with supervisor 4.  

=================================================================

Updated in late May 2023

Another reason we never considered 8.4(2d) was due to CSCwb29379 (see release notes 8.4(2e) below) 

Cisco MDS 9000 Series Release Notes, Release 8.4(2e)

8.4(2e) was available for quite some time now.  So, we upgraded to 8.4(2e) couple weeks ago to fix this problem permanently.  

Verifying that Every EMC Disk Arrays are Properly Attached to an EMC SMI-S Provider

Just update ViPR SRM to point to the new Windows 2016 with SMI-S provider.  However, SMI-S complains it cannot retrieve the info of the array.  Confirm ECOM is running and symcfg list shows the VMAX array.

Restart ECOM service and ViPR SRM can retrieve the stats from the array.   However, I try to find out how to confirm EMC SMI-S does detect the array. 

Follow the kb.
https://www.sentrysoftware.com/kb/KB1132.html

Below is the kb copied from the link. 

Objective

Monitoring EMC disk arrays requires to configure the EMC SMI-S Provider to make sure that the arrays are being discovered. EMC disk arrays are automatically discovered when they are locally attached to the EMC SMI-S Provider. This article explains how to make sure that the EMC SMI-S Provider discovers all disk arrays by using the TestSmiProvider tool.

Procedure

1. Logon to the server where the EMC SMI-S provider is installed
2. Run the TestSmiProvider.exe which can generally be found under “C:\ProgramFiles\EMC\ECIM\ECOM\bin”
   C:\Program Files\EMC\ECIM\ECOM\bin>TestSmiProvider.exe
   Connection Type (ssl,no_ssl) [no_ssl]:
   Host [localhost]:
   Port [5988]:
   Username [admin]:
   Password [#1Password]:
   Log output to console [y|n (default y)]:y
   Log output to file [y|n (default y)]:
   Logfile path [Testsmiprovider.log]:
   Connecting to localhost:5988
   Using user account 'admin' with password '#1Password'

   ########################################################################
   ##                                                                    ##
   ##                  EMC SMI Provider Tester                           ##
   ##   This program is intended for use by EMC Support personnel only.  ##
   ##   At any time and without warning this program may be revised      ##
   ##   without regard to backwards compatibility or be                  ##
   ##   removed entirely from the kit.                                   ##
   ########################################################################
     slp    - slp urls                     slpv    - slp attributes
     cn     - Connect                      dc      - Disconnect
     disco  - EMC Discover                 rc      - RepeatCount
     addsys - EMC AddSystem                remsys  - EMC RemoveSystem
     refsys - EMC RefreshSystem

     ec     - EnumerateClasses             ecn     - EnumerateClassNames
     ei     - EnumerateInstances           ein     - EnumerateInstanceNames
     ens    - EnumerateNamespaces          mine    - Mine classes

     a      - Associators                  an      - AssociatorNames
     r      - References                   rn      - ReferenceNames

     gi     - GetInstance                  gc      - GetClass

     ci     - CreateInstance               di      - DeleteInstance
     mi     - ModifyInstance               eq      - ExecQuery
     gp     - GetProperty                  sp      - SetProperty

     tms    - TotalManagedSpace            tp      - Test pools
     ecap   - Extent Capacity              pd      - Profile Discovery

     im     - InvokeMethod                 active  - ActiveControls
     ind    - Indications menu             tv      - Test views

     st     - Set timeout value            lc      - Log control
     sl     - Start listener               dv      - Display version info
     ns     - NameSpace                    vtl     - VTL menu

   
   

     q      - Quit                         h       - Help
   ########################################################################
   Namespace: root/emc
   repeat count: 1
3. Run the command eq at the prompt
   (localhost:5988) ? eq
   Query Language[DMTF:CQL]:
4. Run the below query
   Query []: SELECT EMC_ArrayChassis.SerialNumber FROM EMC_ArrayChassis

A working provider will return the following for each disk system attached to this provider:

++++ Testing ExecQuery:  ++++

Instance 0:
ObjectPath : //10.0.10.54/root/emc:Clar_ArrayChassis.CreationClassName="Clar_ArrayChassis",Tag="CLARiiON+CKM00083900053"


Clar_ArrayChassis

CLARiiON+CKM00083900053

CKM00083900053

Number of instance qualifiers: 0
Number of instance properties: 3
Property: CreationClassName Number of qualifiers: 0
Property: Tag Number of qualifiers: 0
Property: SerialNumber Number of qualifiers: 0

ExceQuery 1 instances; repeat count 1;return data in 0.000000 seconds

Retrieve and Display data - 1 Iteration(s) In 0.062400 Seconds

A failing provider will return the following:

++++ Testing ExecQuery:  ++++

Error: Connection closed by CIM Server.

Retrieve and Display data - 1 Iteration(s) In 0.053000 Seconds

Please press enter key to continue...

Securely eraseing disk and file

 I used Hiren's Boot Disc and DBAN boot disc to erase my HDD securely most of the time.  I did use dd command but found out there were few other useful commands recommended in this link.  

The first one mentioned in the article is quite useful.  It will overwrite the disk five times and add a final overwrite with zero to hide shredding.

 # shred -n 5 -vz /dev/hda

The command below will use shred and /dev/urandom as the source of random data.

 


shred -v --random-source=/dev/urandom -n1 /dev/sda

 

Another option is to install scrubbing program like scrub in Unix.  You can use it to erase a single file with algorithm.  

scrub -p nnsa|dod|bsi|old|fastold|gutmann|random|random2 fileNameHere

 

Or you can even use it to erase disk with dod standard.  

scrub -p dod /dev/sda

 

*** WARNING: Please test the commands before running it on your data.  Commands mentioned in this article will delete the data permanently.  ***