Lets upgrade to vCenter 6.X: A generic guideline

Lets upgrade our vCenter

*** Updated (22-04-2016): Links changed for vCenter 6.0U2 release.

There are a lot of great blogs out there about the process of upgrading the vCenter from 5.X to the most current version 6.0U2.

Having done the upgrade now in multiple production environments I try to summarize the general process and potential caveats. The following article is more a general approach and not specifically tied to a certain version. I start with the key facts outcome first. The detailed description where those key facts are extracted from are mentioned afterwards.

Key facts

1. Use the vCenter Appliance if possible. Even though there is a fling (unsupported) for a migration from Windows to vCSA I recommend to create it from Scratch at the moment. The upgrades I did with the Appliance went very flawless and is the way to go. Best argument? No need to deal with DBAs anymore ;-).
2. Check all VMware and 3rd party solutions for compatibility.
3. Make sure you can always restore to the current version of vCenter (Database and vCenter backup).
4. Communicate vCenter downtime properly. Make sure which maintenance tasks might be required on 3rd party solution during the upgrade (e.g. stop Horizon View provisioning tasks like refresh on logoff) and that other organizations in the company can deal with the outage. The upgrade can take between 1 hours and 8 hours in case some planning steps haven’t been done accordingly.
5. Have all-mighty DB permissions in place for the VPXD user or have your DBA reachable.
6. As a VMware certified instructor let me advertise the vSphere What’s New 5.5 to 6.X course. It gives you a good overview of the new vCenter architecture, new features and gives you a chance to chat with your instructor about how to plan/process the upgrade. Check it out here

Assessment

Since the vCenter architecture offers a lot of different possibilities (distributed vCenter, vCenter Server Appliance, etc.) I will focus on the following options and evaluate with which option which steps are more important.

• Option 1: Windows based vCenter, all vCenter components installed on the machine, external Microsoft SQL database
• Option 2: vCenter Virtual Appliance (vCVA) with all services embedded (SSO, Webclient, Inventory Service, VPXD) and an embedded database.

Pre-Checks – Version Compatibility List  

The successful upgrade of the vCenter rises and falls with how proper you planned the upgrade. The number of components (in the following called solutions) in our datacenter that communicates with the vCenter is increasing more and more. For every component there must be a check done if it is compatible with the vCenter version you want to upgrade to.

I recommend to maintain the following table (let’s call it the version table) with those information. Try to figure out every solution that is communicating with the vCenter. I typically recommend to a have a dedicated service account for every solution that is connected to the vCenter. In such a case you can easily find all relevant components going through the vCenter login events.

Afterwards go to the Vendor documentation and check for the versions that are compatible for it. For VMware solutions have a look at the following software interoperability matrix.

For every version that needs to be updated make sure if for the current version a direct update to the desired one is possible. One example is if you want to upgrade from vCVA 5.1 to vCSA 6.0U1 you need to be at 5.1 U3 to increase your chance of being successful. If you are on a version < than 5.1U3 you need a hop in between before moving ahead. This information should be included in the version table as well. For VMware solutions, please check the following upgrade path matrix.

Solution name	Current Version	Version compatible with vCenter 6.0U1	Compatible?	Direct Update/upgrade possible?	Comments
vCenter	5.5.0 1476327			yes	Will be part of the upgrade
SSO	2.0			Yes	Will be part of the upgrade. During upgrade SSO will be part of PSC
Operating System (Microsoft)	2012 R2 SP1	2012 R2 SP1	Yes
Microsoft SQL	2012 11.0.5.343 (SP2)	2012 11.0.5.343 (SP2)	Yes
vRealize Automation Center	6.2.1 – 254390	6.2.1 – 254390	Yes
XenDesktop	7.6	7.6	Yes
Citrix Provisionin Server	7.6	7.6	Yes
vCenter SQL native client	10	11	No
Veeam Backup and Replication	8.0.0.2029	8.0.0.2030++	No
Trend Micro Deep Security	9.5	9.6	No
vCloud Network and Security Director	5.5.3	5.5.4. ++	No
Net App Virtual Storage Console Webclient Plugin	6	7	No

Please remember that this document is just an example. Make sure that you have all items verified in the end and include documents/links/screenshots of the related vendor into the comments field.

Verify minimum requirements and dependencies 

1. Check your virtual hardware (*Option 1 only)

Check the amount of virtual hardware for CPU & memory you need for the vCenter after the upgrade. Typically orient yourself on the following table out of the VMware documentation.

For disk space requirements, make sure you have plenty of spare space within the operating system for updates of the OS, helper tools, and the Inventory Service xhive log files that are inflating and deflating a lot. You don’t want any of your vCenter disks running out of space, since this will have a functional impact on your environment. Remember to adjust the pagefile size if you are on a Windows version. You don’t want a 32 GByte pagefile on the disk when you have give 32GByte of memory to your vCenter.

2. Check for synchronous time

Make sure all components are synchronized via NTP or Microsofts internal time synchronizing mechanism (after AD join). An asynchronous time can lead to a lot of strange symptoms throughout your complete environment.

3. Check that all relevant firewall ports are opened / no network service changed has occurred between the versions

From time to time it happens that new services are added that uses other/new network protocols and therefore new network ports. Sometimes service are moved to new from on machine to another (changing an embedded vCSA with vCenter solution and PSC on one machine to a distributed one). Check the documentation or ask the community to figure out what has changed between the versions and what impact your upgrade path might have.

4. Check that you have all relevant accounts and passwords in place

Make sure you have relevant service account passwords (vCenter admin, SSO admin, database user, etc.) in place.

5. Store the correct installation media

Place all relevant files you need for the upgrade on a suitable location. Validate the version numbers.

6. Install relevant / dependant .NET, etc. (* Option 1 only)

Make sure .NET 3.5 SP1 or higher is installed on the machine. Sometime it is not as easy in some environments to get all relevant frameworks into the system where you are planning the upgrade.

7. Communicate downtime for vCenter and validate the influence on your system

Make sure everyone is aware of the outage we will suffer during the vCenter upgrade. Expect a minimum of 60 minutes where no other systems will be able to use the vCenter (Self-service portals, Orchestrator, Backup solutions). So make sure everyone in the connected domains is aware of it and the required processes are running/stopped. Protip: Notify the monitoring guys and make sure they won’t get crazy because of the vCenter outage ;-)

8. Full backup of the vCenter database

To have an application consistent backup of the vCenter database stop the vCenter service and backup the SQL database. It is so important that you are able to restore the vCenter out of nothing. For sure the vCenter upgrade mechanism is doing pre-checks, but nevertheless you might end up in a situation that you can pass the precheck, the upgrade takes place, it uninstalls all current services and BOOOM the installation fails, the database has been altered already and no service is available anymore.

9. Full backup of the vCenter machine (and its components)

See the description above. Backup the whole vCenter VM with any kind of backup solution (Veeam, etc.) and make use of VM snapshots. But remember having the state of the VM repaired doesn’t help you if you have not been able to restore your database.

10. Database permission (* Option 1 only)

That’s something where you can struggle a lot during a vCenter Upgrade. For a DBA you need an incredible large permission set on the database. Even though VMware is defining which permissions are required for which tables, I have never get it work (and many others as well).

The SQL Login needs: DBO on VPXD table, DBO on Systems table AND the permission the following permission on the MSDB table. (Thanks to Alexander Ganser who wrote a lot of findings about it)

GRANT EXCUTE ON msdb.dbo.sp_lock TO VC_ADMIN_ROLE

11. VMware vCenter Upgrade document read and understood

Make sure you have read and understood the document and all implications. If possible do the Upgrade first in an environment similar to your production vCenter. You find the document here

So let’s put those dependencies in a table and send it to your customer or collect all the relevant data before you hop on board to do the upgrade.

Step	Verification	Confirmation	Comments/problems/etc
1	Virtual hardware configured as required
2	Time synchron mechanism configure
3	All Accounts available
4	Installation media transferred
5	Downtime communicated and verified
6	Database backup completed
7	vCenter backup completed
8	Database permission in place
9	vCenter upgraded successful
10	vCenter testplan completed successfully
11	Snapshots and temporary data removed

Plan the upgrade steps

Cool… after around 4 weeks we did all the upgrade pre-checks and have a lot of signed YES’ssss on our Excel-sheets, which means: We are good to go. Now we need to define the order of the Upgrade. In general the upgrade order for VMware solutions is defined in this KB article.

If you have planned everything accordingly, this is going to be the easiest path of your upgrade.

Step	Task	Confirmation	Comments/problems/etc
1	All items in version-table have a yes in the compatible column	yes
2	All dependencies have a yes in the confirmed column	yes
3	Files are accessible
4	Relevant accounts are applicable
5	Communicate downtime
6	Having everything in place for a vCenter recovery (people (DBA) and processes)
7	Upgrade the vCenter		Check the official document first ;-)
8	Verify the Upgrade		Check functionalities, versioning, etc. Might be a separate blogpost necessary
9	Clean Up		After the verification has been done, clean up every item that has been temporarily created (snapshots, backups, installer files, etc)