Install Open VM Tools on Linux VM

Depending on the distro of Linux, I have tough time to install VM Tools on some of them.  The only other option is to install open vmtools.  

Follow VMware instructions to install the open vmtools.

If GUI is required on the VM and assume it is Debian distro, install open-vm-tools-desktop.  

sudo apt-get install open-vm-tools-desktop

Otherwise open-vm-tools will be sufficient.

sudo apt-get install open-vm-tools  

Installing VMware Workstation Pro in OpenSuSE LEAP 15.4

Now system is up and running fine.  Next is to install VMware Workstation Pro.  Basically, it is my OpenSUSE LEAP 15.4 is my host. 

Make sure system is up to date and install the required packages.  Download the binary and copy it to a tmp folder.  Then add the execute permission to the bundle. 

sudo zypper update
sudo zypper install kernel-source kernel-devel gcc

Download the binary and copy it to a tmp folder.  Then add the execute permission to the bundle.   Run the installation after. 

sudo chmod +x VMware-Workstation-Full-16.2.4-20089737.x86_64.bundle
sudo ./VMware-Workstation-Full-16.2.4-20089737.x86_64.bundle

Reboot.  Then I receive the errors below one by one.  

 

  

Do some researches and follow instructions in this video.  

First get the VMware version
vmware -v

Then, download the patches needed to build the VMware host modules against the recent kernel.  You may need to rebuild it if VMware does not start after update.   See https://github.com/mkubecek/vmware-host-modules

wget https://github.com/mkubecek/vmware-host-modules/archive/workstation-16.2.4.tar.gz

tar -xvzf workstation-16.2.4.tar.gz
cd vmware-host-modules-workstation-16.2.4/
tar -cf vmmon.tar vmmon-only
tar -cf vmnet.tar vmnet-only
sudo cp -v vmmon.tar vmnet.tar /usr/lib/vmware/modules/source
sudo vmware-modconfig --console --install-all

I setup a script to start vmware every time the host boots up.  

/etc/init.d/vmware start

I found out none of the USB devices list under Removable devices of the VM.  So, I suspect the vmware-usbarbitrator is not started up auto.  Add the following line to the start up script.  Now, I can see the USB device.

/usr/bin/vmware-usbarbitrator start 

One issue I found and other ppl complain in other Linux distro too.  If you try to copy from VM and paste on the Linux host, you will get error below.

There seems to be no solution to it.  However, if you check your /home/user/.cache/vmware/drag_and_drop/ directory, you will actually find the file there.  

ESX host does not discover new path after migrating to new fabric

Once, I work on fabric migration.  They want to migrate to new equipment and choose not to link the existing and new switch together (no ISL) to have a clean config in new fabric.  The servers connect to different fabric A and B.  On each migration, we need to re-connect storage controller on one fabric and the servers that are connecting the controller to new fabric at the same time.  Only one fabric and one controller will be worked on each time.  Since they are all ESX hosts and no AIX, it is much easier. We complete the job in the weekend during low I/O period.  What happens is after connecting to the new switch, it does not see the new path even VI team rescan.  I did confirm zoning had been completed correctly on the new fabric.  They suggest me to follow VM article Configuring fibre switch so that ESX Server doesn't require a reboot after a zone set change


However, I know RSCN is enabled in the Cisco switch.  Eventually, VI agree to vMotion the VMs in the host, and reboot the ESX host, one by one.  After that, they are discovered correctly. 


One thing we learn, stay away from boot from SAN for ESX host, it takes 30 min for each host to come up after reboot even there are not a lot of luns in the ESX clusters.  It is proven much faster to boot ESX from local disk. 


Also, minimize the number of RDMs!!!  I work in another messy environment with lots of RDM.  Even rescan cannot happen during business hours. 
 

VADP with NetWorker 7.6.x

I have talked about VADP with hotadd in previous posts.  Last month, I have completed testing on VADP with NetWorker 7.6.2.7.  Turn out I should upgrade to NetWorker 7.6.3.7 because of the following two issue.

NW138379 - VADP : Save does not honor the VSS:*=off flag in Windows 2008
This issue is seen if you proxy host is Windows 2008.

Event 16, source: volsnap, Logged: 1/17/2012 6:25:51 PM
The shadow copies of volume E: were aborted because volume, which contains shadow copy storage for this shadow copy, was force dismounted

Another anonying issue is NW139877 - VADP: nsrvadp_save incorrectly reports status as success for aborted backups

I have not tried NetWorker 7.6.4.  However, I have not encountered those issues after upgrading to NetWorker 7.6.3.7.

Whether virtual or physical proxy, you can consult the following page Nuts and bolts in NetBackup for VMware: Transport methods and TCP ports to decide which transport mode you plan to use.  I don't prefer SAN mode even though it is the fastest option.  Presenting a VMFS lun to a Windows 2008 proxy is not an option in some sites.  If you want to use SAN mode, make sure to control who can login to the proxy.  You don't want Windows signature written to a VMFS lun by mistake. 

NetWorker EBR backup error "Internal Cycle Error"

Recently, there was a maintenance done in the VM environment.  All the EBR backup failed.  After rebooting the EBR appliance, we still experience intermittent issue with EBR backup.  Random VM backup failed with error " INTERNAL CYCLE ERROR: backstreamdir failed"

Working with support and eventually find out time is not in sync.  From the logs, they find out start time from NetWorker jobs seem to be about 10s off when other logs are checked from VM and EBR side.  We find out from our VM team time sync is turned off in VMtools for some reason and it should be disabled.  Because our NetWorker server and SN is not in domain, it is pointed to different NTP for time sync.  After confirming time sync issue is fixed, we don't experience any more issue.

So, always go from basic to troubleshoot VM backup like the past, are VMtools up-to-date?  ESX servers running the same version in the VM cluster?  Time is in sync?

NetWorker 8.1.1 EBR VM backup

We deployed NetWorker 8.1 for one of our customer with DD4500 earlier this year.  It works pretty well.  DR and restore are easy.  However, there are a few things to know before you decide to deploy it.

1) EBR and NetWorker server must be resolvable in DNS server (both forward and reverse lookup).  It looks like adding to hosts file is not an option.  Maybe someone want to try hosts file and let me know if they can get it working without DNS.

2) Schedule with override does not seem to work.  For ex, I have schedule with incremental except 1st Fri of the week.  I find out those customized schedules with exception are not available for an "VMWARE action" in the policy.

3) Not enough activity shown in NMC and activity logging in NetWorker.  For ex, when I DR a box, I don't see much info from NMC.  I have to check thru vCenter GUI.

4) If you choose to backup to the internal storage on EBR, you cannot clone them.

5) There is no option to clone on demand.  It is based on policy.  All policy must start as a backup.  So, I cannot just clone them out with nsrclone command.

6) If you plan to backup your VM running SPS, SQL etc, it is not Microsoft application aware backup.  You still need to use NetWorker module for the backup.

7) Need to login to EBR to perform DR.

8) Each proxy can handle only 8 streams.  Deploy more proxy if you have a tighter backup window for your VMs.

packages prerequisite for installing VMtools in CentOS VM

Just image my laptop and building all the VMs again with newer version of OS.  Need to install VMtools on the new CentOS 6 VM but forget the list of packages required.  Find VM kb2002290.

To resolve this issue, run this command before installing VMware Tools:
yum install kernel-devel kernel-headers gcc mkinitrd

Turn off time sync in virtual machine

Recently, I got a request to recover the gst database of NetWorker to retreive some historical backup stats because retention for stats was set too short.  To avoid interfering the production environment, I close the gstd saveset out with FTD as clone device.  Build a Windows 2003 VM on my laptop with same version of NetWorker.  Because retention for stats was too short.  As soon as I recovered the gstd database and started gst service, those expired records would be purged.  Only option is to change the system time of VM to an earlier date within the retention period.  To achieve that, I follow VMware kb1189 to disable the time sync with host.

Since I am using VMware Workstation, I choose to edit the vmx file.  For vSphere, you can add parameters following the kb1189.

Add configuration options in the virtual machine's .vmx file:

Open the virtual machine's configuration file (/vmfs/volumes/datastore_name/vm_name/vm_name.vmx) using a text editor.

1. Set these options to zero. If the entries do not exist, add them.
   
   Note: 0 = disabled, 1 = enabled.
   
   tools.syncTime = "0"
time.synchronize.continue = "0"
time.synchronize.restore = "0"
time.synchronize.resume.disk = "0"
time.synchronize.shrink = "0"
time.synchronize.tools.startup = "0"
time.synchronize.tools.enable = "0"
time.synchronize.resume.host = "0"
   
   Note: In GSX, VMware Fusion, and VMware Workstation, you must use "FALSE" instead of "0". For example:
   
   tools.syncTime = "FALSE"
time.synchronize.continue = "FALSE"
time.synchronize.restore = "FALSE"
time.synchronize.resume.disk = "FALSE"
time.synchronize.shrink = "FALSE"
time.synchronize.tools.startup = "FALSE"
time.synchronize.tools.enable = "FALSE"
time.synchronize.resume.host = "FALSE"

After that, I change the system date, scan in the media.  run recoverpsm and then start gst.  Now, all the stats I need is there. 

SCSI reservation conflict error in ESX

Basically, if you encounter such an error in your ESX environment,

WARNING: SCSI: 5532: Failing I/O due to too many reservation conflicts

WARNING: SCSI: 5628: status SCSI reservation conflict, rstatus 0xc0de01 for vmhba1:0:7. residual R 919, CR 0, ER 3
WARNING: J3: 1970: Error committing txn to slot 0: SCSI reservation conflict

you should follow VMware kb 1005009. 

On the storage side, make sure you follow best practise guide.  For example, array fan in limitation.  For Symmetrix, make sure the front end port is dedicated for ESX connection only and do not mix with other host running other OS. 

Hotadd limitation

Other than the 1TB vmdk limit with early version of VDDK (fixed in VDDK 1.1.1), the following are the limit and suggestion of HotAdd (similar info can be found in vendor support documentation).

1)Hot-Add only works on SCSI disks, not IDE disks
2)Disable automount on proxy host
3)For block size mismatch issue, the VMFS containing the Target VM and the VMFS containing the VADP proxy must use the same or larger block size. For example, if you back up virtual disk on a datastore with 4Mb blocks, the proxy (vRanger) must also be on a datastore with 4Mb blocks or above. Its strongly recomended to install proxy on VM which is residing on 8Mb.

Other consideration not only applied to hotadd but also other transport mode.
1) change the workingDir to a datastore with enough block size if there are more than 1 vmdk for a VM and they are on different datastore with different block size.  For ex, if C drive is on datastore with 1 MB block size, and D drive is on a datastore with 2 MB block size.  workingDir should be on datastore with 2 MB block size.  See VM KB: 1012384

P.S. Do not use hotadd if there is dynamic disk in the VM.  It is just a headache.  See VM KB: 2006279

VADP application quiescing

Recently, I am helping out a friend on VADP deployment.  He has question about VADP quiescing.  I pointed him to an article in Backup Central.  To make sure application quiescing is supported for Windows 2008 R2, make sure ESX 4.1 U1 is deployed (documented in VMware KB: 1031298).  However, if Windows 2008 or Windows 2008 R2 VM deployed under ESX 4.0, make sure KB: 1028881 is followed.

For Exchange and SQL, I still suggest use of Exchange or SQL agent from backup software to backup them up.  When restore is required, for ex, Exchange, you can restore folder in a mailbox easily.

VADP with hotadd configuration

Helper VM is not required for VADP with ESX 4.0 or above if hotadd mode is selected.  Make sure you do have the necessary license for hotadd.  If VCB is still used in the environment, you can follow Commvault's page "Create a Helper Virtual Machine" on how to configure VCB helper VM.

The following is the properties of the proxy VM before snapshot being mounted. 

After snapshot is created, it will be mounted on the proxy host (which is a Virtual machine).  See screenshot below. 

I have backup 2 VM with one disk on each VM at the same time.  That's why you see two extra virtual disk on the proxy host.  Now, you probably understand why you need Advanced version for ESX 4.1 which include the hotadd feature.  Otherwise, you can only use nbd or san transport mode to backup VM. 

Of course, there is limitation on hotadd such as max vmdk size of 1 TB.

Actually, the 1 TB limitation is no longer valid for VDDK 1.1.1 (http://www.vmware.com/support/developer/vddk/VDDK-1.1.1-Relnotes.html) 

If NBD is used, there is a 1 TB limitation.  

VADP / VCB backup failed with hotadd mode

I am testing the VADP backup and keep getting the following error.

"Unable to open a disk of the virtual machine"

Monitor the VC and do see snapshot created.  Run the same backup with nbd and it works fine.  Double check and find out ESX license in on Standard Edition with version 4.1.  

Remember this is a license issue and confirm from VM site VDDK 5.0 release note.

"Licensing. In vSphere 5.0, the SCSI HotAdd feature is enabled only for vSphere editions Enterprise and higher, which have Hot Add licensing enabled. No separate Hot Add license is available for purchase as an add-on. In vSphere 4.1, Hot Add capability was also allowed in Advanced edition. Therefore, customers with vSphere Essentials or Standard edition who use backup products (including VMware Data Recovery) are not able to perform proxy-based backup, which relies on SCSI HotAdd. Those customers must use alternate transport modes."

The most important benefit of using hotadd mode to me is to avoid presenting a VMFS lun to Windows proxy server.

P.S. HotAdd is not supported with VDDK 1.1.1 when proxy server is on ESX 4.1 even correct license is purchased.  See reference link VDDK-1.2.1 release notes

restore to proxy on VADP

With VADP out in 2010, VCB will be out of the picture. However, one thing I don't like about VADP is the lack of restore of the whole image to the proxy host. I know it is not available in NetWorker. When I check the NetBackup guide, I don't see that option too. It will be required to restore directly to ESX host.

Believe it or not, when I work in support, I always discover customer has no space in the SAN for recovery of large VM and they want to keep existing VM. Now, I cannot recover the whole image to proxy and then use VMconverter to move it to a VM Workstation. It has to be recovered to ESX. I will consider it a lack of feature when comparing it with VCB.

VCB with Windows 2008 R2

Recently, I got a question about backing up VM with Windows 2008 R2 as guest.  They mention to me why FULLVM works but File level fails.  Run some test and find out it is caused by the reserve partition configured at the beginning of the 1st Virtual HDD.  Try to assign a drive letter and think it will work.  Nope!  The error indicates it can't find Windows System folder and registry.  As of today, ESX 4.0 U1 release notes indicate it is not supported.  

Conversion wizard of VMware Workstation

One of the benefit of becoming a VCP is to get a free copy of VMware workstation. I finally receive the permanently license key from VMware education. Installing it on my laptop running CentOS. The reason why I use VMware workstation instead of Virtualbox is NetWorking option. In Virtualbox, you cannot set static IP for your VM if the virtual NIC is run under NAT.

With VMware Workstation for Linux, there is no Conversion Wizard. It is only available for Windows version. So, I use the Clone option instead. Starting with VMware 6.5, you can install VMware ESX as VM, so I can get rid of my DL360 now. It is really noisy to run at home.

Installing Solaris 10 64 bits in VMWare Workstation 6.5.0

To install Solaris 10 64 bits, make sure your CPU is 64 bits with VT enable in BIOS.

Even you run the Processor Check for 64-Bit Compatibility and the result is compatible, it may still fail because VT is not enable in BIOS.

For my T400 laptop with Intel 9400, press F1 to goto the BIOS (Config -> CPU).

Enable the following settings
Intel (R) Virtualization Technology
Intel (R) VT-d feature

Save Settings -> Poweroff -> Startup

I have VMWare Workstation 6.5.0 installed on Vista 64 bits Business. When I install Solaris as a guest OS, it will attempt to install as 64 bits. It goes flawlessly. After installation completes and reboots, you should see it start at 64 bits. If you miss that, run isainfo -v (not uname -a).

You can see that this Solaris machine is capable of running 32-bit as well as 64-bit applications.

P.S. The media I use is Solaris 10 update 5. Please check VMWare website for OS support.

ESX 3.0 Patches

The drawback of ESX 3.0 is the lack of patch management. To update patches in ESX 3.0, there is a utility called esxupdate.

Patches for ESX Server 3.x can be applied in any order on ESX Server 3.0 hots with patch ESX-1000074 and ESX Server 3.0.1 hosts with patch ESX-5874303. Later version of ESX Server will include this fix and will not need a patch.

To get a list of all installed updates, execute this command in the ESX Server console:

esxupdate query

Listing Details of Updates:

esxupdate info
esxupdate -l info

Assume NFS installed on a server
mount –t NFSserver:VMBackup /mnt/updates
esxupdate -r /mnt/updates/updatename update (assume updatename is untar)

Installation of VMWare

I try to install VMWare ESX Server on my HP DL360. I got it from ebay for CDN 800 with 4GB of RAM. Installation is very similar to Linux. However, I got a strange error message about checking BIOS settings.

vmkernel: 0:00:00:00.000 cpu0) ALERT: Chipset: 433: no PCI entries in MPS table - check BIOS settings

I remember I change the BIOS settings for OS to Linux. Doing some research on Google and find this VMWare doc. After change the OS back to Windows 2000/2003, the error message disappears.

Details
HP ProLiant servers (and pre-merger Compaq ProLiant servers) require specific BIOS settings in order to be fully compatible with VMware ESX Server. Use of BIOS settings with values other than those identified below might result in operational instability.

If the MPS Table Mode setting within the ProLiant BIOS is not properly configured for ESX Server, the BIOS might not fully populate the PCI interrupt routing entries in the MP Configuration Table (part of the Intel MultiProcessor Specification). An incomplete MP Configuration Table might result in failure of an initial installation of ESX Server, failure to complete the boot process, or instability during normal operation.

Solution
ESX Server requires a BIOS MPS Table Mode setting of Full Table APIC. With the exception of the specific systems referenced below, the following BIOS settings must be applied in order if available:

System Options > OS Selection: Select Windows 2000.
Advanced Options > MPS Table Mode: Select Full Table APIC.

When presented with multiple Windows options (Windows 2000, Windows Server 2003, Windows .NET, and so on) select Windows 2000. If both BIOS settings are available and can be modified, both must be set correctly. You should confirm these settings after any BIOS upgrade operation.

For the HP ProLiant DL 740 and DL 760 servers specifically, set the OS Selection BIOS setting to Linux to allow for proper operation when hyperthreading is enabled. The remainder of the BIOS settings, and the order in which they are applied, are as specified above.

If any BIOS setting is incorrect, you might see one or more messages in /var/log/messages or /var/log/vmkwarning on the Service Console similar to the following:

May 11 22:36:30 esx101 vmkernel: 0:00:00:00.00 ALERT: Chipset: 303: no PCI entries - Check BIOS Settings
May 11 22:36:30 esx101 vmkernel: 0:00:00:00.000 cpu0) ALERT: Chipset: 433: no PCI entries in MPS table - check BIOS settings
May 11 22:36:30 esx101 vmkernel: 0:00:00:00.000 cpu0) WARNING: IOAPIC: 986: no PCI entry for busID 2 busIRQ 8 - trying ISA irq 15

You might also see unnecessary hardware interrupt sharing, which is explained in detail in knowledge base article 1290 at http://kb.vmware.com/kb/1290.

You might need to use the ROM-Based Setup Utility (RBSU), SmartStart CD, or other BIOS configuration tool to make these adjustments. Hewlett-Packard's Web site includes extensive documentation on how to adjust BIOS settings on HP and Compaq systems, including generalized RBSU guides which may be found at http://search.hp.com/query.html?qt=title%3A%22ROM-Based+Setup+Utility+User+Guide%22.

The "OS Selection" section of the HP RBSU User Guide notes that, in certain circumstances, adjusting the OS Selection BIOS setting might result in an automatic change of the MPS Table Mode BIOS setting (emphasis added):

The OS Selection option allows you to select the primary OS for the server. A list of supported OSs for the server displays on the menu, with the ROM defaulting to Microsoft Windows 2000 when the server supports it. Default server Multi Processor (MP) settings and PCI Hot Plug reservation settings are automatically set based on the OS selected.

Although an automatic change might occur, the individual BIOS settings should be confirmed explicitly. For additional assistance in making these changes, please contact HP Technical Support.

Windows 2000 in VMWare Server 1.06

I rebuilt my machine with Windows 2000 and installed VMWare Server 1.06. After that, I would like to create a VM with Windows 2000 for browsing and software testing. After completing installation and reboot, bluescreen comes up with inaccessbile boot device error. I figure out I have setup the HDD with LSI controller. To fix the issue, go to lsilogic website and download the Windows 2000 driver for LSI 20320-R driver. Press F6 when booting from bootable CD / Windows 2000 bootdisk 1. Insert the driver FDD when you are asked. The same applies to XP and Windows 2003.