vMotion Failed – Error: The destination host is not compatible with the hardware version

Today we observed an strange issue with one of our VM where we werw not able to migrate it to another host, actually I was trying to put the ESXi host in maintenance mode to do some activity on host. Ideally when we issue the maintenance mode command it move off all the VMs to other hosts in cluster before going into maintenance mode but in this case it got stuck. We found that one of the VM is not moving to other host automatically, we thought may be there is an anti-affinity rule which is restricting this VM to move over. After checking all the rules we couldn’t find anything, finally we tried to trigger a manual vMotion and then i got this error –  “The destination host is not compatible with the hardware version to which the virtual machine is scheduled to be upgraded. In order to proceed with the operation, virtual machine’s scheduled compatibility upgrade should be disabled.“

The error indicate an issue with the VM hardware version, but after checking the VM we found that it’s  running on latest hardware version v13 (as underlying host is ESXI 6.5) and there is no hardware upgrade scheduled as it’s already running on latest version. We also tried to reboot the server but no luck.

After doing some research we came across a VMware KB article which states –  “This issue occurs if the hardware version upgrade process fails to update the .vmx file.” This make sense in our case since we recently upgraded the hardware version of all of our VMs to v13, you can check my another article how to upgrade the VM hardware version on next server reboot here

Below are the steps motioned in VMware KB to fix this issue – 

1.Power off the virtual machine.
2.Connect to the host on which virtual machine is running using SSH.
3.Navigate to the virtual machine location where the .vmx file is located.
4.Take a backup of the .vmx file.
5.Open the .vmx file using a text editior.
6.Remove these entries related to hardware version upgrade:

virtualHW.scheduledUpgrade.when = “always”
virtualHW.scheduledUpgrade.state = “done”
tools.upgrade.policy = “upgradeAtPowerCycle”

7.Reload the .vmx file for the changes to take effect.

Since above steps needs VM to be powered off and also this is manual work which is very much prone to error. Just out of curiosity i wanted to look at the .vmx file of impacted VM and see what value we have for virtualHW.XXX property which can be fixed remotely. After navigating to the .vmx file I found a NEW state of  vmwareHW.scheduledUpgrade.when  property which is “safe” 

However at GUI we see nothing and it can not be scheduled as it’s already running on latest version. To fix this, I wrote an PowerCLI script using the above resolution provided by VMware but in this case VM does not require any downtime and those manual work on .vmx file. The script will make the require changes on the fly, I tried this to fix my vMotion issue. Here is the code, you can run it on a single server or just provide a list of servers in a text file –

$Servers = Get-Content “c:\server.txt”
foreach ($Server in $Servers)
{
$VM = Get-VM $Server
$VMview = $VM | Get-View
$VMSpec = New-Object VMware.Vim.VirtualMachineConfigSpec
$VMSpec.Tools = New-Object VMware.Vim.ToolsConfigInfo
$VMSpec.Tools.ToolsUpgradePolicy = “manual”
$Spec = New-Object -TypeName VMware.Vim.VirtualMachineConfigSpec
$Spec.ScheduledHardwareUpgradeInfo = New-Object -TypeName VMware.Vim.ScheduledHardwareUpgradeInfo
$Spec.ScheduledHardwareUpgradeInfo.UpgradePolicy = “never”
$Spec.ScheduledHardwareUpgradeInfo.ScheduledHardwareUpgradeStatus = “none”
$VMview.ReconfigVM($VMSpec)
$VMview.ReconfigVM_Task($Spec)
$VMview.Config.ScheduledHardwareUpgradeInfo
}

Note – Before running the script you need to connect to vCenter server by using Connect-VIServer cmdlets.

I hope this has been informative, thanks for reading.