vSphere Update Manager Troubleshooting: Could not scan ESX / ESXi host OR cannot execute upgrade script

Even though I was more involved in conceptual doing in the last months I was recently asked for help since a customer was not able to update their environment via vSphere Update Manager.

The following blog post explains how we can work deal with the following event / task error messages:

“Could not scan ESXihostname” or “Cannot execute upgrade script on host”

The vSphere Update Manager is nowadays integrated in the vCenter Server Appliance and is pretty suitable for patching and upgrading ESXi hosts. In theory the process is really straight forward.

  1. Optional: Create a baseline that includes all relevant ESXi components (so called vSphere Infrastructure Bundles [VIB]) that you want add to your ESXi hosts (e.g. Create a static baseline for a specific vSphere build)
  2. Attach the baseline to a Cluster or an ESXi host object
  3. Scan the Cluster or ESXi host object against the baseline. In the end we can discover which baseline elements are installed on the ESXi host and which are missing. In case all baseline items exist on the ESXi host the host is declared as comply.
  4. Remediate the baseline. The ESXi host will be placed into the maintenance mode, the components defined in the baselines will be installed on the ESXi host. The host will reboot and the maintenance mode will be disabled. Afterwards the ESXi host should be comply to the baseline.

That is quite easy. Easy enough that this module takes around 1 hour in the vSphere: Install, Configure & Manage class (from time to time I still deliver VMware Trainings -> contact me if you want to join :).

The problem: In production environment from time to time things become more complicated as in the labs. Especially the usage of custom ESXi images by certain vendors (e.g. HPE) might create some complexity here.

Before you read on: Verify that DNS is correctly set up. vCenter must be able to lookup the ESXi DNS names (forward) and the IP address to the ESXi DNS names (reverse).

Besides that, verify that the time of the vCenter Server Appliance and the ESXi hosts match the system time.

If you still got problems, two log files can help us gathering more information about the root cause of the problem. Navigate to the ESXi host via SSH or to the Shell.

ESXi: /var/log/vua.log

vCenter Server Appliance: /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server-log4cpp.log

Attaching the baseline to the cluster and clicking on Check compliance will check if all hosts match to the baseline.

In case no result is shown and error are thrown within tasks and events we need to have a look at the proper log file.

Checking the /var/log/vua.log we can figure out what is causing the malfunctional upgrade script execution

In most cases this is related some sort of conflicting VIB. In our case there seems to be a payload difference between the VIB ELX**** on our system.

Since we don’t use any EMULEX adapters (which drivers are part of the ELX VIB) we can remove it. Place the ESXi host in maintenance mode so that any operation done has no impact on Virtual Machines.
Please be aware of that if you are having a vSAN Cluster virtual machine components can still reside on the ESXi host even though it is in maintenance mode.

Find the proper VIB name by using esxcli software vib list | grep elx

and remove it via esxcli software vib list remove -n VIBNAME.

ATTENTION: Please make sure that you don’t actively use one of the affected VIBs.

After the VIB has been removed reboot the ESXi host. and redo the update manager compliance check.

In case the scan is still not working with a task message:

VMware vSphere Update Manager had an unknown error. Check the events log files

I would recommend to have a look at the vCenter Server Appliance /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server-log4cpp.log

Before spending too much time on troubleshooting it. Take a snapshot of the vCenter Server Appliance and reset the Update Manager database. Connect via SSH to the vCenter Server Appliance and change to the bash.

>shell

///changed to bash

> service-control --stop vmware-updatemgr
> /usr/lib/vmware-updatemgr/bin/updatemgr-util reset-db
> rm -rf /storage/updatemgr/patch-store/*
> service-control --start vmware-updatemgr

Reconfigure your Update Manager settings (conenctivity and co and create your baselines). My friend Alex Ganser has explained that issue in a far more detailed level. Have a look!

2 thoughts on “vSphere Update Manager Troubleshooting: Could not scan ESX / ESXi host OR cannot execute upgrade script

  • 25. April 2019 at 20:10
    Permalink

    We recently went through this. After removing that vib the host will report as non-compliant with VUM’s default critical baseline. If you remediate against that baseline it’ll reinstall the ELX vib and put you back at square one. We had to create a custom baseline that excludes this VIB along with an exclusion for “complete update 2” which the machines already have anyway.

    Reply
    • 25. April 2019 at 20:19
      Permalink

      Good to know! I have not realized that since as explained in the text I had created specific baseline for the current build. I will add a note to the text. Thanks a lot

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

Time limit is exhausted. Please reload CAPTCHA.

This site uses Akismet to reduce spam. Learn how your comment data is processed.