vCenter 6.7 Update 1 - Converge to embedded failed!

Update 27th October 2018: I was meant to provide this update last weekend, but it has been a hectic week. Within a few hours of posting this issue, a few people at VMware reached out to me to understand more about my configuration, asked me to upload the logs and a few of the smart guys and gals at VMware had been able to reproduce the issue. As it turns out, the key in the APPLMGMT_PASSWORD vecs store is used when a file based backup of the VCSA is scheduled. I believe a KB is on the way, but the workaround is to delete the VCSA backup schedule, perform the converge and you can then re-enable the schedule after the converge is complete.

Update 31st March 2019: This has been out for a little while but I forgot to update this post … here is a link to the VMware KB article for this issue.

So today I was having a play in the lab with the new vcsa-util utility in vCenter 6.7 Update 1, that provides the ability to converge an external PSC and vCenter Server in to an embedded deployment. I’ve got a walkthrough post on working through this in a basic environment coming soon, stay tuned.

Anyhow, I’d gone ahead and taken my backups, snapshots, got the pre-reqs in place, got my JSON files ready and then I went ahead and ran to the tool to start the converge process.

.\vcsa-util.exe converge --backup-taken --no-ssl-certificate-verification --verbose C:\vcsaconverge\sitea-vc01-converge.json


After a couple of minutes, the process finished but unfortunately I saw a big FAILED message in the output even though the Precheck validations has succeeded:

[01/18] [SUCCEEDED] Precheck validations for converge
[FAILED]Gather requirements
Task 'MonitorPSCDeployTask: Running MonitorPSCDeployTask' execution failed because [Error occurred while doing converge operation!], possible resolution is []
================================================================================
Converge to embedded failed!
C:\Users\matt.LAB\AppData\Local\Temp\vcsaCliInstaller-2018-10-18-05-56-pvhd7cue\workflow_1539842201048\sitea-vc01-converge\converge_mgmt.log
because [ERROR: Converge Process Failed!], possible resolution is []
================================================================================
Error occurred. See logs for details.
Exception message: com.vmware.vcsa.installer.converge.monitor_psc_deploy: ERROR:
Converge Process Failed! Trace:


As you can see, there is nothing really helpful here, especially the “possible resolution is []”, but I guess that’s what you get when you run a brand new tool that has been out for a day or so as well.

After getting some time later in the evening when they family were in bed, I had a look at the converge_mgmt.log file which is exported to a temporary folder on the machine you are running vcsa-util from (the exact folder is in the output above). Inside of this log file, I found the following error message:

2018-10-18T10:13:53.664Z INFO converge Store all vecs certificates.
2018-10-18T10:13:53.926Z ERROR converge Failed to get vecs users and permissions. Error: {
"resolution": null,
"problemId": null,
"componentKey": null,
"detail": [
{
"id": "install.ciscommon.command.errinvoke",
"translatable": "An error occurred while invoking external command : '%(0)s'",
"localized": "An error occurred while invoking external command : 'Command: ['/usr/lib/vmware-vmafd/bin/vecs-cli', 'entry', 'getcert', '--store', 'APPLMGMT_PASSWORD', '--alias', 'location_password_default', '--output', '/root/velma/old_certs/APPLMGMT_PASSWORD.crt']\nStderr: Error: No certificates were found for entry [location_password_default] of type [Secret Key].\nvecs-cli failed. Error 87: Operation failed with error ERROR_INVALID_PARAMETER (87) \n'",
"args": [
"Command: ['/usr/lib/vmware-vmafd/bin/vecs-cli', 'entry', 'getcert', '--store', 'APPLMGMT_PASSWORD', '--alias', 'location_password_default', '--output', '/root/velma/old_certs/APPLMGMT_PASSWORD.crt']\nStderr: Error: No certificates were found for entry [location_password_default] of type [Secret Key].\nvecs-cli failed. Error 87: Operation failed with error ERROR_INVALID_PARAMETER (87) \n"
]
}
]
}
2018-10-18T10:13:53.936Z INFO converge Cleanup successful with partial flag = True.


I took the “args” section of that output and look at the command that was trying to be run, which ended up being the following:

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store APPLMGMT_PASSWORD --alias location_password_default


I opened an SSH session to the vCenter Server I was trying to converge and ran the above command, and sure enough I got the same output that was being shown in the converge_mgmt.log file:

[email protected] [ ~ ]# /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store APPLMGMT_PASSWORD --alias location_password_default
Error: No certificates were found for entry [location_password_default] of type [Secret Key].


I then used vecs-cli to list the entries in this specific store, getting the response below:

[email protected] [ ~ ]# /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store APPLMGMT_PASSWORD
Number of entries in store :    1
Entry type :    Secret Key
Certificate :


So I could see there was not a certificate in the store, but there was a key.

Now, I do not know what this store is used for, so if you know I’d love if you can leave me a comment below, or reach out to me on twitter @mattallford. I checked a couple of other 6.7 environments, and a few 6.5 ones, and also spun up a clean external 6.7U1 environment in my lab, and this store was empty on all of them. So provided this was a lab environment, I decided to try and remove the entry from the store on the vCenter Server using the following command:

/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store APPLMGMT_PASSWORD --alias location_password_default