Backup and restore ESXi via PowerCLI after a failed boot device

A few days ago I encountered the following error in the webclient of my homelab.

lost_connectivity_to_the_device_mpx

"Lost connectivity to the device mpx.vmhba32:C0:T0:L0 backing the boot filesystem /vmfs/devices/disk/mpx.vmhba32 :C0:T0:L0. As a result, host configuration changes will not be saved to persistent storage."

In a production environment you would now check the events of your out-of-band management component (IMM, iLO, DRAC, etc.) for a failed SD-card or USB boot device. In such a situation a reboot of the ESXi host (e.g. after doing some updates) wouldn’t be a good idea. To fix the problem we need to …

  1. replace the boot device
  2. reinstall ESXi
  3. reconfigure ESXi (in an automated PowerCLI way)

The first two steps can usually be done pretty quickly. The configuration of the ESXi can take a little bit more time and is more error prone when it’s done manually. That’s why we are going to use the good old ESXi embedded backup feature (formerly called esxcfg-backup on the Shell) with the PowerCLI cmdlet Get– and Set-VMHostFirmware.

Open a PowerCLI session to your vCenter and run the following snippet.

$output = 'C:\TEMP\'
Get-VMHost $esxiName | Get-VMHostFirmware -BackupConfiguration -DestinationPath $output

Verify that a file is created in the output path in format: configBundle-ESXiFQDN.tgz

now that you have created a backup-config file of the ESXi you can shutdown the ESXi, fix/replace the SD-Card and reinstall ESXi (I recommend using the same ESXi build otherwise the import of the backup file might not work properly).

Once the ESXi host installed, configure it with the former management network settings (IP, GW, DNS, hostname), add it to your vCenter and put the ESXi into maintenance mode.

Use the PowerCLI session to run the following lines.

$esxiName = 'ESXiFQDN*' # Use the same ESXi name as in the webclient OR keep the *
$input = 'C:\Temp\configBundle-ESXiFQDN.tgz'
$esx = Get-VMHost $esxiName
$esx | Set-VMHostFirmware -Restore -Force -SourcePath $input

Wait a few seconds and reboot the ESXi afterwards. It should come back online with the identical settings as before the boot device failure.

You can find the Powershell files right here in my GitHub repository.

Happy recovery!

2 thoughts on “Backup and restore ESXi via PowerCLI after a failed boot device

  • 3. March 2016 at 5:05
    Permalink

    Hi Guys

    I have similar error warning appears on Esxi 5.5.0,1746018.

    “Lost connectivitya to the device mpx.vmhba32:C0:T0:L0 backing the boot filesystem /vmfs/devices/disks/mpx.vmhba32:C0:T0:L0, As a result, host configuration changes will not be saved to persistent storage”

    Hardware : HP ML350 G6 .

    I don’t have V center license. is there any other way to reconfigure Esxi on new USB disk ?

    Cheers
    Kash

    Reply
  • 15. March 2016 at 23:01
    Permalink

    I think it would not work because it seems that you have the ESXi free edition. In this edition the API is locked for set/write commands.

    One way to achieve the backup would be to go to ESXi shell (direct or SSH) and use esxcfg-backup.

    Please be aware of that you are backing up the ESXi and no virtual machine itself.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

Time limit is exhausted. Please reload CAPTCHA.