Backup and restore ESXi via PowerCLI after a failed boot device

A few days ago I encountered the following error in the webclient of my homelab.

lost_connectivity_to_the_device_mpx

"Lost connectivity to the device mpx.vmhba32:C0:T0:L0 backing the boot filesystem /vmfs/devices/disk/mpx.vmhba32 :C0:T0:L0. As a result, host configuration changes will not be saved to persistent storage."

In a production environment you would now check the events of your out-of-band management component (IMM, iLO, DRAC, etc.) for a failed SD-card or USB boot device. In such a situation a reboot of the ESXi host (e.g. after doing some updates) wouldn’t be a good idea. To fix the problem we need to …

  1. replace the boot device
  2. reinstall ESXi
  3. reconfigure ESXi (in an automated PowerCLI way)

The first two steps can usually be done pretty quickly. The configuration of the ESXi can take a little bit more time and is more error prone when it’s done manually. That’s why we are going to use the good old ESXi embedded backup feature (formerly called esxcfg-backup on the Shell) with the PowerCLI cmdlet Get– and Set-VMHostFirmware.

Open a PowerCLI session to your vCenter and run the following snippet.

$output = 'C:\TEMP\'
Get-VMHost $esxiName | Get-VMHostFirmware -BackupConfiguration -DestinationPath $output

Verify that a file is created in the output path in format: configBundle-ESXiFQDN.tgz

now that you have created a backup-config file of the ESXi you can shutdown the ESXi, fix/replace the SD-Card and reinstall ESXi (I recommend using the same ESXi build otherwise the import of the backup file might not work properly).

Once the ESXi host installed, configure it with the former management network settings (IP, GW, DNS, hostname), add it to your vCenter and put the ESXi into maintenance mode.

Use the PowerCLI session to run the following lines.

$esxiName = 'ESXiFQDN*' # Use the same ESXi name as in the webclient OR keep the *
$input = 'C:\Temp\configBundle-ESXiFQDN.tgz'
$esx = Get-VMHost $esxiName
$esx | Set-VMHostFirmware -Restore -Force -SourcePath $input

Wait a few seconds and reboot the ESXi afterwards. It should come back online with the identical settings as before the boot device failure.

You can find the Powershell files right here in my GitHub repository.

Happy recovery!

4 thoughts on “Backup and restore ESXi via PowerCLI after a failed boot device

  • 3. March 2016 at 5:05
    Permalink

    Hi Guys

    I have similar error warning appears on Esxi 5.5.0,1746018.

    “Lost connectivitya to the device mpx.vmhba32:C0:T0:L0 backing the boot filesystem /vmfs/devices/disks/mpx.vmhba32:C0:T0:L0, As a result, host configuration changes will not be saved to persistent storage”

    Hardware : HP ML350 G6 .

    I don’t have V center license. is there any other way to reconfigure Esxi on new USB disk ?

    Cheers
    Kash

    Reply
  • 15. March 2016 at 23:01
    Permalink

    I think it would not work because it seems that you have the ESXi free edition. In this edition the API is locked for set/write commands.

    One way to achieve the backup would be to go to ESXi shell (direct or SSH) and use esxcfg-backup.

    Please be aware of that you are backing up the ESXi and no virtual machine itself.

    Reply
  • 21. July 2020 at 20:57
    Permalink

    Thanks for a very nice article. I’m in a situation where I need to reinstall ESXi to resolve some issues. It was originally installed on 2 Gig SD cards in RAID 1. I have an issue with Update Manager and this host. I cannot patch the host anymore. I also want to insert larger SD cards, so I thought it might make sense to backup the config, upgrade the SD cards, reinstall ESXi, and then import the config. Backing up the config is easy. Reinstalling ESXi should be easy enough. How can I ensure it is at the same build level as before so that the config will restore properly? I have heard that if the build is different, the configuration won’t restore. Does the force switch help with that during the restore?

    Also, according to your article, you suggest reconnecting the host to vcenter before restoring the config rather than after. I would have thought the config needs to be restored first. Does it matter? If I can reconnect it to vcenter, then maybe I can use Update Manager to patch the host to the same level or even a higher level before the config restore? Again, how critical is it to be at the precise same build level? What is the Update Manager restores it to a slightly newer level?

    Reply
    • 11. September 2020 at 10:57
      Permalink

      It has been a while since I used this mechanism. Please let me know if you figured something new out here. From my experience it was critical that the same build was used; afterwards for sure everyone could be updated/upgraded via the update manager.

      Reply

Leave a Reply to Jonathan Shapiro Cancel reply

Your email address will not be published.

Time limit is exhausted. Please reload CAPTCHA.

This site uses Akismet to reduce spam. Learn how your comment data is processed.