Even though I was more involved in conceptual doing in the last months I was recently asked for help since a customer was not able to update their environment via vSphere Update Manager.
The following blog post explains how we can work deal with the following event / task error messages:
“Could not scan ESXihostname” or “Cannot execute upgrade script on host”
The vSphere Update Manager is nowadays integrated in the vCenter Server Appliance and is pretty suitable for patching and upgrading ESXi hosts. In theory the process is really straight forward.
- Optional: Create a baseline that includes all relevant ESXi components (so called vSphere Infrastructure Bundles [VIB]) that you want add to your ESXi hosts (e.g. Create a static baseline for a specific vSphere build)
- Attach the baseline to a Cluster or an ESXi host object
- Scan the Cluster or ESXi host object against the baseline. In the end we can discover which baseline elements are installed on the ESXi host and which are missing. In case all baseline items exist on the ESXi host the host is declared as comply.
- Remediate the baseline. The ESXi host will be placed into the maintenance mode, the components defined in the baselines will be installed on the ESXi host. The host will reboot and the maintenance mode will be disabled. Afterwards the ESXi host should be comply to the baseline.
That is quite easy. Easy enough that this module takes around 1 hour in the vSphere: Install, Configure & Manage class (from time to time I still deliver VMware Trainings -> contact me if you want to join :).
The problem: In production environment from time to time things become more complicated as in the labs. Especially the usage of custom ESXi images by certain vendors (e.g. HPE) might create some complexity here.
Before you read on: Verify that DNS is correctly set up. vCenter must be able to lookup the ESXi DNS names (forward) and the IP address to the ESXi DNS names (reverse).
Besides that, verify that the time of the vCenter Server Appliance and the ESXi hosts match the system time.
If you still got problems, two log files can help us gathering more information about the root cause of the problem. Navigate to the ESXi host via SSH or to the Shell.
vCenter Server Appliance: /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server-log4cpp.log
Attaching the baseline to the cluster and clicking on Check compliance will check if all hosts match to the baseline.
In case no result is shown and error are thrown within tasks and events we need to have a look at the proper log file.
Checking the /var/log/vua.log we can figure out what is causing the malfunctional upgrade script execution
In most cases this is related some sort of conflicting VIB. In our case there seems to be a payload difference between the VIB ELX**** on our system.
Since we don’t use any EMULEX adapters (which drivers are part of the ELX VIB) we can remove it. Place the ESXi host in maintenance mode so that any operation done has no impact on Virtual Machines.
Please be aware of that if you are having a vSAN Cluster virtual machine components can still reside on the ESXi host even though it is in maintenance mode.
Find the proper VIB name by using esxcli software vib list | grep elx
and remove it via esxcli software vib list remove -n VIBNAME.
ATTENTION: Please make sure that you don’t actively use one of the affected VIBs.
After the VIB has been removed reboot the ESXi host. and redo the update manager compliance check.
In case the scan is still not working with a task message:
VMware vSphere Update Manager had an unknown error. Check the events log files
I would recommend to have a look at the vCenter Server Appliance /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server-log4cpp.log
Before spending too much time on troubleshooting it. Take a snapshot of the vCenter Server Appliance and reset the Update Manager database. Connect via SSH to the vCenter Server Appliance and change to the bash.
///changed to bash
> service-control --stop vmware-updatemgr
> /usr/lib/vmware-updatemgr/bin/updatemgr-util reset-db
> rm -rf /storage/updatemgr/patch-store/*
> service-control --start vmware-updatemgr
Reconfigure your Update Manager settings (conenctivity and co and create your baselines). My friend Alex Ganser has explained that issue in a far more detailed level. Have a look!