vCenter 6.5: #SRM, #vSphere Replication, #NSX problems after SSL change (LS_Update_certs.py)

After upgrading to vCenter 6.5 I replaced the Certificate Authority certificate of my (external) Platform Service Controller (PSC) with an ‘flenzquest-enterprise ;-)’ signed certificate.

The tasks to replace the ssl certificates haven’t changed much from version 6.0 and has been document very well within the community.

After the successful replacement I realized that I had problems with vSphere replication and NSX. I know that NSX is not supported yet with vSphere 6.5, but so far the NSX Manager connectivity with vCenter 6.5 has worked pretty well (until I replace the certificates).

I had a very bad feeling about this issue and googling about it brought an old case to my attention which I thought has been fixed quite a while ago (obviously it hasn’t). I found an old chat protocol of me, Frank Büchsel and Feidhlim O’Leary on Twitter.

chat_ssl

I have blogged about the fix on my old blog vXpertise.net before it was hacked and unfortunately that was one of the posts we lost in the old days. Even though the problem has been described by VMware in KB2121701 many problem have and had struggled with the workaround (especially using the python script LS_Update_certs.py resulting in Updated 0 service (s)).

Since the KB is written pretty detailed and well I recommend to get an high-level solution thinking with my post and afterwards dive into the deep-physical stuff of the KB.

Understand the problem

First of all: Understand the problem. I am not sure if that problem still occurs because of my concrete situation (upgrade from vCSA 5.5 -> 6.0 U1/U2 -> 6.5, might have used different certificates in the past,etc.)

It might be technically not 100% correct, but my abstraction and explanation level should be enough to understand what the problem is and how it is fixed.

When we roll-out a new vCenter Server Appliance and PSC, the certificates are generated by the Certificate Authority of the Platform Service Controller. Since no one know this guy, no one really trusts him, therefore its certificates are marked red in the browser (and my graphic).

At the same time we need to know that on the PSC a service called lookup service is running. I think about the lookupservice like a good old phone directory. The lookupservice knows other solutions connected to it and has some data about them (how to reach them via https://URL, how to know them: SSL_Certificate of the solution)

You can gather the current lookupservice data via

https://psc01.lenzker.local/lookupservice/mob?moid=ServiceRegistration&method=List

So the following model, show how it should look like in a default setup.

orig_certficates

So what’s happening when we replace the certificates of our solutions like vCenter and PSC? Everything works fine, but the lookupservice is not updating its entries. A solution that is verifying if there is a match between the actual solution certificate and the data within the lookupservice will now fail -> NSX, SRM, vSphere Replication.

certificates_broken

How to fix it

Take a Browser and go to

https://psc01.lenzker.local/lookupservice/mob?moid=ServiceRegistration&method=List

login as SSO-Admin (like administrator@vsphere.local), remove the content between <filtercriteria> click on invoke method and voila,

screen-shot-2016-11-28-at-22-02-09you see all registered solutions.

screen-shot-2016-11-28-at-22-03-31

Use the inspect element icon, copy the whole base64 certificate

Copy the data into an empty text-file. Call the file psc_old.cer (make sure file extensions are shown) and extract the SSL-thumbprint (we will need this later). Copy the Thumbprint in an empty text file

ssl_thumbprint

Now the tough part will beginn.

Transfer the current chain.pem file of the PSC, Sub-CA, Enterprise-CA to your PSC, e.g. with winSCP into the /root folder

The format of the chain file must be Base64 encoded and look like this:

—–BEGIN CERTIFICATE—–
MIID2jCCAsKgAwIBAgIJANxwO561hXjWMA0GCSqGSIb3DQEBCwUAMGMxCzAJBgNV
BAYTAkRFMRAwDgYDVQQIEwdCYXZhcmlhMQ8wDQYDVQQHEwZNdW5pY2gxEDAOBgNV
....SOLUTION CERTIFICATE
5ox8bjyyTKKPgoFZOFkLZhpaA5yVxH1JI7VEKS67XBzK5Gg0IuXqFmr2rwWZmTLg
AnNKXrkBHJmbH67hv/q3BVrYF3XvFKyrHHlNIrTX
—–END CERTIFICATE—–
—–BEGIN CERTIFICATE—–
MIIDgzCCAmugAwIBAgIQNOUNUInl57hNKSDxI0lrWTANBgkqhkiG9w0BAQUFADBI
MRUwEwYKCZImiZPyLGQBGRYFbG9jYWwxFzAVBgoJkiaJk/IsZAEZFgdsZW56a2Vy
SUB CA CERTIFICATE
6/flzPsPJsGsQGHf1nPBCOBLCzwDymZnlZxoKRTwrYKxV+cYKFRaJl4ioogK+DbF
LIoYWgC6biYcV0s0f/GgsoFOAminHUrLDzDDWxK5xloviPIiNZyP
—–END CERTIFICATE—–

—–BEGIN CERTIFICATE—–
MIIDgzCCAmugAwIBAgIQNOUNUInl57hNKSDxI0lrWTANBgkqhkiG9w0BAQUFADBI
MRUwEwYKCZImiZPyLGQBGRYFbG9jYWwxFzAVBgoJkiaJk/IsZAEZFgdsZW56a2Vy
…  CA CERTIFICATE
6/flzPsPJsGsQGHf1nPBCOBLCzwDymZnlZxoKRTwrYKxV+cYKFRaJl4ioogK+DbF
LIoYWgC6biYcV0s0f/GgsoFOAminHUrLDzDDWxK5xloviPIiNZyP
—–END CERTIFICATE—–

Make sure no whitespace appear and no empty line are included (use a good editor like Notepad++ or Sublime).

Now we need to connect to the PSC via SSH and navigate in the following folder:

cd /usr/lib/vmidentity/tools/scripts/

Run the following line (as documented in the KB). Important:

  • Remove the whitespaces of the SSL Thumbprint
  • Type the complete command manually into SSH shell. Do not copy and paste. I was not able to get the whole thing running when I was copy and pasting anything. Type every char of the thumbprint and doublecheck. If you fail here the legendary Updated 0 service (s)) will occur. It seems that the script is searching for the SSL certificate with the thumbprint we have put into the command. If for any reason the thumbprint is not matching any more (because previous attempts succeeded with a wrong certificate), check my graphics, try to understand what you must do and repeat the steps
  • Do Snapshots of the PSC/vCSA before you do anything

ls_update_certs.py –url https://psc01.lenzker.local/sdk –fingerprint ‎d79531d1dc743ba43cbaebb735b8bf1aa139a168 –certfile /root/chain.cer –user administrator@vsphere.local –password VMware99!

works fine (I know I pasted a password, but I just want to make sure you know that you not necessarly need to to deal with special chars in the password).

screen-shot-2016-11-28-at-20-42-08

Check that multiple services have been updated.

Repeat the steps for vCenter Server Appliance

The following graphic should show you the final state.

certificates-fix

Enjoy SRM, NSX, vSphere Replication and many more.

Before:

screen-shot-2016-11-28-at-19-22-44

After:

screen-shot-2016-11-28-at-20-42-33

5 thoughts on “vCenter 6.5: #SRM, #vSphere Replication, #NSX problems after SSL change (LS_Update_certs.py)

  • Pingback: vSphere 6.5 Link-O-Rama » Welcome to vSphere-land!

  • 7. February 2017 at 19:42
    Permalink

    Hi Fabian,

    Just a quick Update on this with vSphere 6.5, now the funny part is that you have to enter the fingerprint with “:” so the command on vSphere 6.5 have to look like this one:

    python ls_update_certs.py –url https://psc.testdrive.local/lookupservice/sdk –fingerprint 6A:3D:41:79:D0:70:5C:GB:54:F6:51:C9:8A:9A:C6:9E:4A:79:16:BE –certfile /var/tmp/VMCA.crt –user Administrator@vsphere.local –password VMware1!

    Reply
  • Pingback: vCenter 6.5 : #SRM, #NSX, #vSphere Data Protection, #vSphere Replication problems after SSL Change (ls_update_certs.py) – Toby Systems

  • 6. April 2017 at 16:27
    Permalink

    My problem is executing: /usr/lib/vmware-vmca/bin/certificate-manager

    Error while reverting certificate for store : MACHINE_SSL_CERT
    Rollback Status : 0% Completed [Rollback operation failed]

    Error while performing rollback operation, please try Reset operation…

    Never get out from here… No matter what values I fill in generating the new certs

    Reply
  • 6. July 2017 at 21:00
    Permalink

    I’ve been getting the “0 services updated” and was wondering what the heck I was doing wrong. Then I stumbled on your blog…

    Apparently there is a stupid invisible character from the thumbprint I copied into the command. I noticed it when my cursor paused just before the thumbprint while pressing the right arrow key in notepad++.

    I removed the invisible character and then all of my prepared python commands were copy-pastable.

    Thank you!

    Reply

Leave a Reply to Tobias P. Cancel reply

Your email address will not be published.

Time limit is exhausted. Please reload CAPTCHA.

This site uses Akismet to reduce spam. Learn how your comment data is processed.